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ABSTRACT 


We  verified  the  USAF  World  Wide  Merged  Cloud  Analysis  (WWMCA)  against 
observations  from  the  Cloudsat  atmospheric  sounder.  We  analyzed  WWMCA  data  for 
2010  for  two  regions  that  are  of  high  interest  to  the  national  intelligence  community 
and  that  differ  in  their  meteorological  characteristics.  The  two  regions  covered  were: 
(a)  much  of  southwest  Asia;  and  (b)  much  of  western  Russia  and  the  Barents  Sea.  We 
analyzed  WWMCA  performance  according  to  four  criteria:  (1)  type  of  cloud  event 
(Definite-Cloud,  Probable-Cloud,  and  No-Cloud);  (2)  geographic  region;  (3)  time  of  day; 
and  (4)  time  of  year.  We  measured  WWMCA  performance  using  contingency  table 
metrics  and  found  marked  differences  in  performance  for  the  four  criteria.  In  particular, 
WWMCA  tended  to  perform  better  in  analyzing:  (a)  No-Cloud  and  Definite-Cloud  events 
than  Probable-Cloud  events;  (b)  the  lower  latitude  region  than  the  higher  latitude  region; 
and  (c)  persistent  cloud  events  than  variable  cloud  events.  Our  Heidke  skill  scores 
indicated  that  WWMCA  performance  was,  in  general,  moderately  better  than  that  of  a 
random  set  of  analyses.  Overall,  WWMCA  performance  was  problematic,  given  that 
WWMCA  is  a  near  real-time  analysis  product  and  is  designed  to  initiate  short  lead  time 
cloud  forecasts  used  by  the  intelligence  community. 
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I.  INTRODUCTION 


A.  IMPORTANCE  OF  CLOUD  ANALYSES  AND  FORECASTS 

Cloud  analysis  and  forecasts  are  essential  to  the  overhead  imagery  mission. 
Imagery  satellites  play  a  vital  role  in  the  national  intelligence  mission.  They  are  low 
density/high  demand  assets  that  are  quite  sensitive  to  cloud  masking  of  targets.  Cloud 
cover  impacts  satellite  imagery  collection  by  obscuring  sensors  operating  in  the  visual 
and  infrared  (IR)  channels.  Accurate  cloud  forecasts  assist  in  target  selection,  the  asset 
scheduling,  and  help  reduce  the  need  to  reshoot  targets.  To  improve  cloud  forecasts,  a 
firm  verification  of  the  cloud  analyses  that  go  into  cloud  forecasts,  and  of  the  forecasts 
themselves,  is  needed. 

Cloud  analysis  verification  is  very  difficult,  because:  (1)  clouds  can  form  and 
dissipate  in  a  relatively  short  period;  and  (2)  clouds  reside  at  various  levels  in  the 
atmosphere  and  are  advected  by  winds  via  nonlinear  processes.  Satellites  observations 
can  provide  constant  coverage  over  much  of  Earth  and  their  images  can  be  merged  into  a 
single,  two-dimensional  mosaic  depicting,  for  example,  cloud  top  height  for  a  given  area. 
However,  this  does  not  provide  the  three-dimensional  (cloud  thickness)  analysis  of  clouds 
necessary  to  meet  operational  requirements  for  creating  accurate  cloud  forecasts. 

Currently,  the  Air  Force  Weather  Agency  (AFWA)  is  using  the  Cloud  Depiction 
Forecast  System  (CDFS)  II  to  merge  satellite  imagery  and  conventional  (surface  and 
upper  air  soundings)  observations  into  a  global  cloud  analysis  product  called  Worldwide 
Merged  Cloud  Analysis  (WWMCA).  WWMCA  initiates  the  cloud  forecast  models  that 
provides  cloud  forecasts  to  the  national  intelligence  community  for  overhead  imagery 
collection  planning.  To  date,  there  has  not  been  an  extensive  study  to  verify  operational 
CDFS  II  products  against  independent  observations  (the  studies  that  have  been  done  are 
described  later  in  this  chapter).  The  WWMCA  component  of  CDFS  II  is  difficult  to 
verify  because  there  are  few  independent  instruments  designed  to  detect  clouds  that  have 
global  coverage  that  are  not  used  in  developing  WWMCA  products  (i.e.,  few  satellite 
observations  that  are  not  ingested  into  CDFS  II). 
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Our  goal  in  this  study  was  to  begin  the  testing  of  CDFS  II  by  verifying  the 
operational  cloud  analysis  products  generated  by  WWMCA.  Our  main  research 
questions  were: 

1.  How  should  WWMCA  products  be  verified? 

2.  How  well  does  WWMCA  perform  according  to  the  type  of  cloud  condition,  or 
cloud  event? 

3.  How  well  does  WWMCA  perform  according  to  the  time  of  day  and  the  time 
of  year? 

4.  How  well  does  WWMCA  perform  according  to  geographic  location  or 
region? 

B.  SCOPE  OF  THIS  RESEARCH 

The  purpose  of  this  research  was  to  develop,  test,  and  apply  a  method  to  verify 
WWMCA  products  against  cloud  information  from  a  space-based  radar  system,  Cloudsat. 
Additionally,  we  wanted  to  develop  initial  verification  results  for  use  in  evaluating  the 
impacts  of  CDFS  II  products  on  the  operational  planning  and  operational  outcomes  of  the 
IC  users  of  CDFS  II  products — that  is,  to  develop  information  that  can  be  used  to  answer 
the  questions  “How  do  cloud  analysis  and  forecast  products  affect  satellite  imagery 
collection?  Figure  1  provides  an  overview  of  the  approach  being  used  in  research  at  the 
Naval  Postgraduate  School  (NPS)  to  address  these  operational  impacts.  The  upper  left 
portion  of  this  figure  summarizes  the  approach  used  in  this  study — comparing  cloud 
analyses  to  cloud  observations  to  verify  the  analyses.  In  future  studies,  we  will  compare 
cloud  forecasts  to  cloud  observations  to  verify  the  forecasts.  The  upper  right  portion  of 
Figure  1  summarizes  the  approach  being  used  in  a  companion  study  at  NPS  that  is 
verifying  the  impacts  of  overhead  imagery  mission  planning  on  mission  outcomes 
through  the  use  of  a  mission  planning  and  outcomes  model.  The  bottom  portion  of 
Figure  1  indicates  how  the  results  from  the  two  types  of  verification  studies  will  be 
merged  to  quantitatively  assess  the  impacts  of  cloud  analyses,  cloud  forecasts,  and  actual 
clouds  on  operational  planning  and  operational  outcomes. 
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Analyzing  Cloud  Analysis  and  Forecast  Impacts  on  Imagery  Collection 


Verification  of  Cloud  Analyses  and 


Forecasts 


Verification  of  Operational  Plans 


Metrics  of  the  Impacts  of  Cloud 
Analyses,  Cloud  Forecasts,  and 
Actual  Clouds  on  Operational 
Performance 


Figure  1.  A  schematic  illustration  of  the  approach  being  used  at  NPS  to  quantitatively 
assess  the  impacts  of  cloud  analyses,  cloud  forecasts,  and  actual  clouds  on 
operational  planning  and  operational  outcomes.  The  upper  left  portion  of  the 
figure  summarizes  the  approach  being  used  in  this  study  to  verify  WWMCA 
cloud  analyses.  The  upper  right  portion  of  the  figure  summarizes  the 
approach  being  used  in  a  companion  NPS  study  to  verify  operational 
planning.  The  bottom  portion  of  the  figure  represents  the  merger  of 
information  from  the  two  verification  studies. 


C.  PRIOR  RESEARCH 

The  most  relevant  prior  studies  for  our  research  were  conducted  by  the  University 
Corporation  for  Atmospheric  Research  (UCAR),  Atmospheric  Environmental  Research, 
Inc.  (AER),  and  the  U.S.  Air  Force  16th  Weather  Squadron.  The  first  was  a  WWMCA 
verification  study  in  which  WWMCA  output  was  compared  against  Cloudsat  data 
(UCAR  2008),  and  the  second  was  a  small  follow  up  effort  performed  in  2010  (AER 
2010).  Details  of  these  studies  are  available  to  the  United  States  government  personnel 
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and  contractors  but  were  never  captured  in  a  technical  report  or  paper  and  are  only  briefly 
summarized  in  a  verification  and  validation  paper  by  Northrop  Grumman  Information 
Systems  (NGIS  2011).  The  studies  covered  two  periods  spanning  approximately  three 
months:  01  April — 29  June  2008  (Northern  Hemisphere  only)  and  28  March — 31  May 
2010  (Northern  and  Southern  Hemispheres).  These  studies  concluded  that  WWMCA 
detected  clouds  when  Cloudsat  said  there  was  cloud  64%  of  the  time  in  the  Northern 
Hemisphere  and  more  than  80%  of  the  time  in  the  Southern  Hemisphere.  NGIS  (2011) 
reported  that  WWMCA  tended  to  under  analyze  cloud,  as  opposed  to  falsely  assigning 
cloudy  conditions  to  no-clouds  conditions. 

The  study  by  the  16th  Weather  Squadron  (Selin  2011,  personal  communication), 
was  a  continuation  of  the  UCAR  study  but  used  Cloudsat  data  to  assess  WWMCA 
performance  in  identifying  clouds  at  low,  middle,  and  high  levels  in  the  Northern 
Hemisphere.  Two  three  month  periods  were  evaluated:  01  April — 29  June  2008  and 
28  March — 31  May  2010.  The  study  concluded  that  WWMCA  does  an  adequate  job  of 
classifying  low  and  middle  clouds  correctly,  if  it  is  able  to  detect  them.  WWMCA  had 
the  hardest  time  detecting  low  and  middle  clouds.  Selin  suggests  that  problems  in  the 
detection  of  low  and  middle  level  clouds  is  probably  the  biggest  obstacle  when  it  comes 
to  using  WWMCA  for  cloud  level  and  cloud  profile  information. 

Ruggiero  (2000)  describes  a  study  conducted  by  the  Air  Force  Research 
Laboratory  (AFRL)  to  examine  processes  for  using  cloud  analysis  products  to  initialize 
numerical  weather  prediction  (NWP)  models.  The  study  focused  on  eastern 
Massachusetts  in  September  1995  and  used  surface  observations  and  rawinsonde  data  in 
the  verification  process.  The  study  concluded  that  the  cloud  detection  algorithms  used  in 
WWMCA  at  the  time  of  the  study  were  able  to  detect  clouds  when  clouds  were  present 
81%  of  the  time  and  agreed  with  the  observed  cloud  fraction  73%  of  the  time  (NGIS 
2011). 

Horsman  (2007)  verified  WWMCA  against  real-time  surface  weather 
observations  at  ten  different  Air  Force  bases  within  the  continental  United  States  for 
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16  days.  This  study  found  that  WWMCA  performance  varied  by  climate  region. 
Overall,  the  study  found  that  overall,  WWMCA  performed  poorly  with  a  “verification  of 
27%  and  a  miss  rate  of  32%”  (Horsman  2007). 

Norquist  (2007)  verified  WWMCA  against  field  measurements  collected  from 
cloud  profiling  radar  (CPR)  and  a  portable  lidar  during  two  efforts  at  Hanscom  Air  Force 
Base,  MA.  From  36  days  of  data,  a  total  of  117  hours  were  selected  in  which  both 
instrumental  observations  and  WWMCA  were  available  (by  design,  cloud  cover  was 
present  in  all  observations).  “Of  these  1 17  hours,  WWMCA  detected  clouds  in  91  giving 
WWMCA  an  overall  cloud  detection  rate  of  78%”  (NGIS  2011). 

Gustafson  et  al.  (2011)  compared  Moderate  Resolution  Imaging 
Spectroradiometer  (MODIS)  derived  cloud  mask  data  from  Aqua  and  Terra  to  WWMCA 
cloud  cover  data  for  June  and  September  of  2010.  Cloud  fractions  for  both  MODIS  cloud 
masks  and  WWMCA  were  calculated  and  classified  as  either  clear  (<20%),  partly-cloud 
(20 — 80%),  and  cloudy  (>80%).  Table  1  displays  the  comparison  results  from  Terra  for 
June  2010  (Aqua  results  were  very  similar).  Overall,  the  cloud  fractions  of  WWMCA 
and  MODIS  were  in  agreement  65%  of  the  time.  WWMCA  performed  less  well  in  polar 
regions  and  relatively  well  in  oceanic  glint  regions  and  over  bright  backgrounds,  such  as 
deserts.  Gustafson  concluded  that  MODIS  cloud  mask  data  is  less  than  ideal  as  an 
independent  source  of  cloud  truth. 

Table  1.  The  results  of  cloud  fraction  binning  from  WWMCA/MODIS  (Terra) 
comparison  from  June  2010.  Table  based  on  NGIS  (201 1). 


Cloud  Fractions 

Percentage  (%) 

Clear  matched 

25 

Partly  cloudy  matched 

6 

Cloudy  matched 

34 

Bartlett  (2009)  evaluated  several  methods  for  verifying  WWMCA  cloud 
distributions.  These  methods  include  comparisons  with  MODIS  imagery  and  cloud- 
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related  information  from  the  Navy  Operational  Global  Atmospheric  Prediction  System 
(NOGAPS).  The  study  concluded  that  quantitative  verification  of  cloud  distributions  can 
be  achieved  and  that  more  reference  distributions  based  on  satellite  imagery  be 
developed. 

In  2004  and  2011,  the  14th  Weather  Squadron  and  AER,  looked  at  the  long-term 
mean  behavior  of  WWMCA  (NGIS  2011).  Their  study  concluded  that  the  merging 
processes  used  to  develop  WWMCA  products  can  introduce  errors  that  are  artifacts  of  the 
merging  processes.  Most  of  the  artifacts  were  found  at  the  higher  latitudes  where 
observations  are  provided  primarily  by  polar  orbiting  satellites  (NGIS  2011).  The  study 
also  concluded  that  some  of  WWMCA’ s  performance  problems  arise  from  physical 
inconsistencies  in  the  merging  processes. 

It  is  difficult  to  summarize  the  results  of  these  prior  studies,  since:  (a)  they  are 
based  on  different  periods,  locations,  and  analysis  methods;  and  (b)  most  are  not  well 
documented.  However,  these  studies  seem  to  indicate  that:  (1)  the  WWMCA  probability 
of  detection  (POD)  for  clouds  may  be,  overall,  in  the  range  of  65-85%;  (2)  WWMCA 
seems  to  under  analyze  clouds,  especially  highly  optically  thin  cirrus  (in  part  by  design) 
and  low  clouds  over  oceans;  and  (3)  verification  of  WWMCA  is  problematic  due  to  the 
relative  lack  of  independent  and  comprehensive  verifying  data.  These  prior  studies 
assessed  WWMCA  products  from  a  wide  range  of  years.  However,  CDFS  II  has 
undergone  many  changes  since  it  was  first  implemented  in  2002.  There  have  been 
multiple  science  level  changes  and  the  tunable  parameters  have  been  continuously  altered 
to  improve  the  baseline  performance.  Thus,  the  results  from  the  prior  studies  are  useful 
but  may  be  out  of  date,  or  soon  may  become  out  of  date. 

The  prior  studies  have  provided  useful  results,  but  they  have  a  number  of 
important  shortcomings  that  indicate  further  studies  are  needed.  These  shortcomings 
include: 

1.  The  data  sets,  methods,  and  results  for  these  studies  have  not  been  well 
documented. 

2.  The  studies  covered  relatively  short  periods. 
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3.  Most  of  the  studies  only  looked  at  small  regions. 

4.  The  studies  that  investigated  large  regions  provided  their  results  at 
relatively  coarse  spatial  resolutions. 

5.  The  studies  suffered  from  a  lack  of  comprehensive,  independent 
observational  data  against  which  to  verify  WWMCA. 

6.  The  studies  apparently  used  and/or  reported  only  limited  verification 
metrics  (e.g.,  POD  was  reported  but  not  FAR,  bias,  Heidke  skill  score,  or 
other  metrics  needed  to  develop  a  full  verification). 

7.  The  studies  provided  little  information  on  how  WWMCA  performance  has 
changed  over  time,  as  upgrades  to  CDFS  II  have  been  made. 

To  address  these  shortcomings  we  designed  our  study  to  achieve  the  following 
objectives. 

1.  Design  and  develop  a  method  to  do  routine,  near  real-time  verification  of 
WWMCA  based  on  Cloudsat  observations. 

2.  Test  the  method  for  a  range  of  cloud  events,  locations,  and  periods. 

3.  Develop  initial  verification  results  using  a  range  of  metrics. 

4.  Develop  initial  verification  results  for  use  in  evaluating  the  impacts  of 
cloud  analyses,  cloud  forecasts,  and  actual  clouds  on  the  planning  and 
outcomes  of  satellite  imagery  collection  operations.  That  is,  provide 
initial  information  for  answering  the  question:  “How  do  cloud  analysis  and 
cloud  forecast  performance,  and  actual  clouds  themselves,  affect  the 
planning  and  outcomes  of  satellite  imagery  collection  operations?” 

D.  ORGANIZATION  OF  THESIS 

Chapter  II  gives  background  information  on  CDFS  II,  WWMCA,  and  Cloudsat, 
and  also  discusses  the  methodology  used  to  verify  and  compare  WWMCA  data  using 
Cloudsat.  Chapter  III  discusses  the  results  of  the  WWMCA  verification.  Chapter  IV 
provides  a  study  summary,  conclusions,  and  recommendations  for  future  research. 
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II.  DATA  AND  METHODS 


A.  OVERVIEW 

This  chapter  provides  background  information  on  CDFS  II,  WWMCA,  Cloudsat, 
and  Cloudsat’ s  data  product  2B-GeoProf.  The  datasets  used  for  this  study  and  the 
methodology  for  processing  and  matching  WWMCA  and  Cloudsat  data  are  discussed. 

B.  CDFS  II  AND  WWMCA  DESCRIPTION 

1.  Cloud  Depiction  Forecast  System  (CDFS)  II 

The  Cloud  Depiction  Forecast  System  (CDFS)  II  is  a  computer  processing  system 
designed  and  operated  by  the  Air  Force  Weather  Agency  (AFWA)  that  produces  an 
hourly,  global,  cloud  analysis  from  individual  satellites  and  merges  them  with 
conventional  observations  (e.g.,  surface  observations  and  upper  air  soundings)  into  a 
Worldwide  Merged  Cloud  Analysis  (WWMCA)  that  reports  the  following  parameters 
(HQ  AFWA/DNXM  2011): 

•  total  cloud  percentage 

•  layered  cloud  percentage 

•  layered  cloud  type 

•  layered  cloud  top  heights 

•  layered  cloud  base  heights 

•  pixel  mean  time  in  Julian  minutes 

Figure  2  illustrates  the  different  CDFS  II  components,  including  WWMCA,  how  they  are 
related  to  each  other,  and  the  process  for  generating  cloud  analyses  and  forecasts.  Both 
polar  orbiting  and  geostationary  satellites  provide  observations  to  CDFS  II.  The  polar- 
orbiters  are  those  in  the  Defense  Meteorological  Satellite  Program  (DMSP)  and  National 
Oceanic  and  Atmospheric  Administration  (NOAA)  Polar  Orbiting  Environmental 
Satellites  (POES)  program.  The  geostationary  satellites  include  those  in  the 
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Geostationary  Operational  Environmental  Satellite  (GOES)  and  the  European  Space 
Agency’s  Meteorological  Satellite  (METEOSAT)  programs.  The  satellite  and 
conventional  observations  undergo  a  four  level  process  to  be  merged  into  a  global  cloud 
analysis.  Level  one  is  data  calibration,  level  two  classifies  each  pixel  into  cloudy  or 
clear,  level  three  applies  cloud  layering  and  typing,  and  level  four  consists  of  merging  the 
separate  analyses  into  one  global  analysis,  called  WWMCA.  WWMCA  is  used  to  initiate 
a  suite  of  cloud  forecast  models  (HQ  AFWA/DNXM  2011).  Refer  to  Appendix  A  for  a 
detailed  description  of  the  four  processing  levels. 


Inputs  Analysis 

1  Defense  Meteorological  Satellite  Program 

2  Polar  Operational  Environment  Satellite 

3  Geostationary  Operational  Environment  Satellite 

4  Meteorological  Satellites  operated  by  European  Space  Agency 

5  Worldwide  Merge  Cloud  Analysis 


Output  Forecast 

6  Forecaster  in  the  loop 

7  Advect  Cloud 

3  Stochastic  Cloud  Forecast  Model  (Global) 

9  Diagnostic  Cloud  Forecast  (Regional) 


Figure  2.  Schematic  overview  of  the  Cloud  Depiction  Forecast  System  II. 

Observational  data  are  received  from  meteorological  satellites,  conventional 
observations,  and  global  analyses  from  various  models.  These  data  are 
merged  into  one  global  cloud  analysis,  called  WWMCA,  which  is  used  to 
initiate  cloud  forecast  models.  Quality  control  is  performed  on  the  cloud 
analysis  and  model  output  products  by  a  forecaster.  Figure  from  HQ 

AFWA/DNXM  (2011). 


WWMCA  is  known  to  have  some  deficiencies  (HQ  AFWA/DNXM  2011).  First, 
WWMCA  often  places  high  cirrus  clouds  at  too  low  an  altitude  due  to  surface 
radiation  emission  contamination  in  the  brightness  temperatures.  Second,  CDFS  II  tends 
to  under  analyze  low  clouds.  A  correction  is  required  because  low  cloud  top 
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temperatures  may  be  very  close  or  exceed  that  of  the  underlying  surface  temperatures, 
thus  producing  a  false  colored  pixel  (HQ  AFWA/DNXM  201 1). 

2.  Worldwide  Merge  Cloud  Analysis  (WWMCA) 

WWMCA  provides  information  for  a  global  grid  of  square  cells  that  each  covers 
an  area  of  24  km  by  24  km  (true  at  60°  latitude).  For  each  cell,  the  following  information 
is  provided: 

•  i  and  j  spatial  coordinates 

•  total  cloud  percentage 

•  layered  cloud  percentage 

•  layered  cloud  type 

•  layered  cloud  top  heights 

•  layered  cloud  base  heights 

•  mean  pixel  time  in  Julian  minutes 

Figure  3  shows  the  data  format  of  a  WWMCA  file.  The  total  cloud  amount  is  the 
percentage  of  cloud-filled  pixels  divided  by  the  by  the  total  number  of  pixels  per  cell. 
Layered  cloud  amount  uses  the  same  calculation  as  total  cloud  amount,  except  that  the 
calculation  uses  the  coldest  (top)  layer  pixels.  Cloud  base  height  is  calculated  using  the 
RTNEPH  (Real  Time  Nephanalysis)  technique  (Kiess  and  Cox  1988).  Cloud  top  height 
is  calculated  by  interpolation  of  the  Naval  Operational  Global  Atmosphere  Prediction 
System  (NOGAPS)  atmospheric  temperature  and  height  profiles  to  the  mean  cloud  top 
temperatures  for  each  grid  cell.  The  highest  percentage  of  cloud  type  in  the  final  analysis 
decides  cloud  type  for  the  cell  (HQ  AFWA/DNXM  2011).  WWMCA  is  used  to 
initialize  three  cloud  forecast  models: 

1.  Advect  Cloud  (ADVCLD) 

2.  Stochastic  Cloud  Forecast  (SCFM) 

3.  Diagnostic  Cloud  Forecast  (DCF) 
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Figure  3.  Worldwide  Merge  Cloud  Analysis  file  format.  Figure  from  HQ 

AFWA/DNXM  (2011). 


C.  CLOUDSAT 

1.  Description  of  Cloudsat 

Cloudsat  is  a  National  Aeronautics  and  Space  Administration  (NASA)  operated 
satellite  launched  on  April  28,  2006.  It  carries  a  W-band  (94  GHz)  cloud  profiling  radar 
(CPR)  for  sensing  condensed  cloud  particles  and  detecting  precipitation,  and  is  part  of  the 
Earth  Systems  Science  Pathfinder  (ESSP)  mission.  The  original  Cloudsat  program  was 
funded  for  22  months  and  has  been  extended  to  September  2011.  Ground  operations  are 
performed  by  the  United  States  Air  Force  (USAF)  at  Kirkland  Air  Force  Base  in 
Albuquerque,  New  Mexico.  The  CPR  has  been  providing  continuous,  global  time  series 
of  vertical  cloud  structure  and  properties  at  a  vertical  resolution  of  485  m  since  June  2, 
2006  (Stephens  et  al.  2008). 

Cloudsat  was  designed  to  help  fulfill  four  mission  objectives:  “(1)  quantify  the 
representation  of  clouds  and  cloud  processes  in  global  atmospheric  circulation  models, 
(2)  quantify  the  relationships  between  vertical  profiles  of  cloud  liquid  water  and  ice  and 
radiative  heating  of  the  atmosphere  and  surface;  (3)  evaluate  cloud  properties  retrieved 
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from  other  satellites  system  (e.g.,  Aqua);  and  (4)  contribute  to  improving  the 
understanding  of  the  indirect  effect  of  aerosols  on  clouds  by  investigating  the  effect  of 
aerosols  on  cloud  and  precipitation  formation”  (Stephens  et  al.  2008),  which  is  critical  to 
the  understanding  of  climate  change. 

The  Cloudsat  satellite  flies  in  a  constellation  of  satellites  referred  to  as  an  “A- 
Train,”  which  consists  of  the  Cloud-Aerosol  Lidar  and  Infrared  Pathfinder  Satellite 
Observation  (CALIPSO)  satellite,  followed  by  other  NASA  satellites  (Figure  4). 
CALIPSO  has  a  two-wavelength,  polarization-sensitive  backscattering  lidar  that  provides 
high  resolution  vertical  profiles  of  clouds  (NASA  Facts-Cloudsat  2003).  Cloudsat  uses 
near-nadir  pointing  millimeter-wavelength  radar  capable  of  probing  the  vertical  structure 
of  a  cloud  (Cloudsat  Data  Products  Handbook  2011).  The  combination  of  data  from 
theses  satellites  provides  a  valuable  source  of  cloud  information.  The  other  satellites  that 
makeup  the  “A-Train”  are  Aura,  Polarization  and  Anisotropy  of  Reflectances  for 
Atmospheric  Sciences  coupled  with  Observations  from  a  Lidar  (PARASOL),  Aqua,  and 
Orbiting  Carbon  Observatory  (OCO). 


Figure  4. 


Depiction  of  the  satellites  that  make  up  the  “A-train”.  Each  satellite 
has  its  name  and  time  of  equator  crossing  listed.  Note  that  CALIPSO  trails 
Cloudsat  by  15  seconds  to  allow  for  synergy  between  Aqua,  Cloudsat,  and 
CALIPSO.  From  NASA  Facts-A-Train  2003. 
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The  constellation  flies  in  a  Sun-synchronous  orbit  with  a  mean  equatorial  altitude 
of  705-730  km  and  an  inclination  of  98.2°.  This  orbit  is  fixed,  so  that  there  are  no 
changes  in  the  orbital  elements  over  long  time  periods.  The  revisit  time  is  16  days.  This 
means  that  Cloudsat  repeats  ground  track  every  16  days,  or  233  revolutions  (Cloudsat — 
Education:  FAQ  2011). 

Clouds  are  weak  scatters  of  microwave  radiation,  in  contrast  to  the  stronger 
reflections  from  Earth’s  surface.  To  detect  these  weak  cloud  signals,  the  CPR  was 
designed  for  a  minimum  detectable  signal,  Zmin.  of  -28  dBZ  (Stephens  et  al.  2008).  Table 
2  summarizes  the  defining  parameters  of  the  radar  confirmed  by  measurements  before 
and  after  launch.  The  CPR  emits  a  3.3  microsecond  pulse  resulting  in  a  vertical 
resolution  of  485  m.  The  back  scattered  signal  is  oversampled  to  produce  a  range  gate 
spacing  of  240  m.  From  the  altitude  of  705  km,  the  instantaneous  field  of  view  (IFOV), 
at  mean  sea  level,  is  1.7  km  along  and  1.3  km  across  track  (Figure  5),  and  688  pulses  are 
averaged  to  produce  a  nominal  footprint  of  2.5  km  along  track.  The  volume  defined  by 
the  along  track  footprint  and  240  m  range  is  referred  to  as  a  range  resolution  volume 
(RRV)  (Mace  et  al.  2007). 


Table  2.  Cloud  profiling  radar  (CPR)  parameters  and  their  respective 

performances.  From  Stephens  et  al.  2008. 


Parameter 

Performance 

Frequency 

94.05  GHz 

Altitude 

705-730  km 

Range  resolution  (6  dB) 

485  m 

Cross-track  resolution 

1.4  km 

Along-track  resolution 

1.8  km 

Pulse  width 

3.3  (js 

Peak  power  (measured) 

32.6  dB 

PRF 

3700-4300-  Hz 

Antenna  diameter 

1.85  m 

Antenna  gain 

63.1  dBi 

Antenna  side  lobes 

-50  dB  @  0  >  7° 

Integration  time  (single-beam) 

0.16  s 

Data  window 

30  km 

Minimum  detected  reflectivity  (measured) 

-30  dBZ 
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Figure  5.  The  instantaneous  field  of  view  (IFOV)  for  the  Cloudsat  cloud  profiling  radar 

(CPR).  From  Cloudsat  Product  Handbook  (201 1). 

Due  to  the  radiometric  differences  between  cold  scenes  (e.g.,  clear  skies  over  a 
cold  surface  at  night)  and  hot  scenes  (e.g.,  clear  skies  over  a  warm  surface  during 
daytime),  the  single-beam  Zmjn  varies  by  ±  1  dB.  Cloudsat’s  calibration  as  of  2008  of 
Zmin  is  29.9  to  30.9  dBZ  (Stephens  et  al.  2008). 

Cloudsat  has  problems  detecting  some  low  level  strata,  cumulus,  non-drizzling 
stratocumulus,  warm  altocumulus  composed  of  small  water  droplets,  and  optically  thin, 
high  cirrus  (Mace  et  al.  2007).  Since  the  CPR  vertical  resolution  is  485  m  and  surface 
reflectivity  is  up  to  five  times  greater  than  cloud  reflectivity,  measurements  in  the  lowest 
two  RRVs  of  the  profile  are  dominated  by  surface  back  scatter.  This  surface 
contamination  can  extend  up  to  1  km,  and  the  signal  usually  returns  to  maximum 
sensitivity  by  the  5th  RRV.  However,  on  average,  clouds  that  reflect  above  10  dBZ  in  the 
3ld  and  4th  RRVs  (i.e.,  480  to  960  m  above  surface)  are  detected  above  surface 
contamination  (Mace  et  al.  2007).  Sassen  and  Khvorostyanov  (2007)  explain  how  the 
radiative  and  backscattering  properties  of  mixed-phase  clouds  effect  their  detection. 

2.  Description  of  2B  Cloud  Geometrical  Profiling  (GeoProf)  Product 

The  2B  Cloud  Geometrical  Profiling  Product  (GeoProf)  identifies  the  levels  in  the 
vertical  column  sampled  by  Cloudsat  that  contain  significant  radar  echo  from 
hydrometeors  and  provides  an  estimate  of  the  radar  reflectivity  factor  for  each  of  these 
volumes.  Details  on  the  Geoprof  algorithms  are  provided  in  the  Level  2  GEOPROF 
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Product  Process  Description  and  Interface  Control  Document  (2007).  The  2B  GeoProf 
product  uses  an  algorithm  that  reads  the  Cloudsat  1B-CPR  granule,  which  measures  the 
backscatter  power  as  a  function  of  distance  from  the  radar,  and  the  corresponding 
MODIS-AUX  and  ECMWF-AUX  granules  to  produce  the  cloud  mask  (Cloudsat 
Standard  Data  Products  Handbook  2008).  MODIS’  cloud  fraction  from  the  visible  250  m 
MOD35  product  is  integrated  over  the  Cloudsat  footprint  (CloudSat  2B  GEOPROF 
Quality  2007). 

The  cloud  mask  data  is  stored  in  the  2B -GeoProf  data  product  and  contains  a 
value  between  0  and  40  for  each  range  bin,  with  values  greater  than  5  indicating  the 
likelihood  of  hydrometeors  (Stephens  et  al.  2008).  Larger  values  indicate  a  higher 
likelihood  of  hydrometeors  and  a  lower  likelihood  of  false  detections  (Table  3).  A 
Cloudsat  data  file  contains  one  granule,  where  a  granule  is  the  data  collected  during  one 
orbit.  Granules  are  broken  into  31  segments  that  allow  for  a  visual  view  of  the  data 
(Figure  6). 


Table  3.  Description  of  Cloudsat  cloud  mask  values,  false  detection  rates,  and  false 
detections.  The  percentage  of  false  detection  is  100  times  the  number  of 
false  detections  divided  by  the  total  number  of  detections  for  the  specific 
cloud  mask  value.  From  Stephens  et  al.  (2008). 


Mask  Value 

Meaning 

%  false  detections  goal 

Estimated  %  false  detection 
via  CALIIPSO  comparison 

-9 

5 

6-10 

Bad  or  missing  radar  data 

Significant  return  power  but  likely  surface  clutter 
Very  weak  echo  (detected  using  along-track 

<50% 

44% 

20 

averaging) 

Weak  echo  (detection  may  be  artifact  of  spatial 

<16% 

5% 

30 

correlation) 

Good  echo 

<21% 

4.3% 

40 

Strong  echo 

<0.2% 

0.6% 
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Figure  6.  A  visualization  of  Cloudsat  data  for  one  segment  of  a  granule.  The  top  pane  is 
the  1A  Aux  data  and  contains  the  date  and  start  time  of  the  granule,  and  the 
starting/ending  latitude-longitude  and  times  of  that  segment.  The  middle  panel 
is  the  2B-GeoProf  cloud  masking,  which  is  color  coded  for  masking  values. 
The  bottom  panel  is  an  Aqua  MODIS  image  (11  pm)  of  the  granule,  which 
was  taken  one  minute  prior  to  the  data  collection  of  the  granule.  From 
Cloudsat  Data  Processing  Center  (201 1). 


D.  STUDY  REGIONS  AND  PERIODS 

We  selected  two  regions  for  this  study  based  on  open  source  information  about 
where  the  U.S.  national  intelligence  community  has  a  high  interest  in  electro-optical  (EO) 
imagery.  We  selected  WWMCA  Box  22  (Figure  7)  because  it  encompasses  much  or  all 
of  Iraq,  Iran,  Afghanistan,  and  Pakistan,  all  of  which  are  important  in  U.S.  national 
security.  WWMCA  Box  22  contains  mountains,  deserts,  ocean,  and  coastal  regions  and 
mostly  resides  in  the  mid-latitudes,  extending  south  to  18°N  and  north  to  50°  N,  and 
within  the  field  of  view  (FOV)  for  geostationary  satellites.  The  diversity  of  climates, 
terrains,  and  the  availability  of  geostationary  satellite  information  for  use  in  developing 
WMCA  products,  makes  Box  22  a  good  case  for  testing  the  mid-latitude  performance  of 
WWMCA.  We  also  selected  WWMCA  Box  29  (Figure  7)  because  it  too  covers  areas  of 
interest  to  the  IC.  Box  29  encompasses  much  of  western  Russian  and  the  Barents  Sea  in 
the  Arctic,  a  region  of  increasing  interest  due  to  decreases  in  Arctic  sea  ice  and  the 
potential  for  increased  Arctic  maritime  activity.  Box  29  extends  between  50°  N  and 
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90°  N,  so  WWMCA  products  for  this  box  rely  on  polar  orbiting  satellites.  Thus,  Box  29 
is  a  good  case  for  assessing  the  high  latitude  performance  of  WWMCA. 

The  period  of  the  study  was  January,  April,  July,  and  October  2010.  These 
months  were  chose  to  represent  the  four  main  seasons  of  the  year.  We  limited  our  study 
to  just  two  WWMCA  boxes,  and  four  months  of  one  year  due  to  time  constraints. 
However,  the  methods  developed  in  our  study  can  be  readily  applied  to  assess  the 
performance  of  many  other  boxes,  months,  and  years.  They  can  also  be  readily  applied  to 
look  at  regions  smaller  or  larger  than  one  WWMCA  box,  and  to  periods  less  than  or 
greater  than  one  month.  See  Chapter  IV  for  more  information  on  applications  of  methods 
and  extensions  of  our  study. 


Figure  7.  Location  of  WWMCA  boxes  22  and  29,  shown  by  the  white  southern  and 
northern  boxes,  respectively.  These  boxes  appear  skewed  in  the  image  due  to 
the  map  projection.  Box  22  includes  much  or  all  of  Iraq,  Iran,  Afghanistan, 
and  Pakistan.  Box  29  includes  much  of  western  Russia  and  the  Barents  Sea. 
Base  image  from  Google  Earth  (201 1). 
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E. 


DATA  SETS 


1.  WWMCA 

The  WWMCA  data  set  used  for  this  study  was  provided  by  the  16th  Weather 
Squadron  and  consisted  of  global,  hourly  cloud  analyses  for  all  of  2010  in  an  ASCII 
format.  The  file  name  convention  is:  the  prefix  CLD,  year,  month,  and  WWMCA  box 
number.  For  example,  the  data  files  for  WWMCA  Box  22  for  March  and  October  2010 
are  CLD201003022  and  CLD201010022,  respectively.  The  data  files  contain  all  the 
hourly  cloud  analyses  for  a  WWMCA  box  for  an  entire  month  and  have  about  1.5  million 
lines  of  data  per  file.  One  zipped  WWMCA  data  file  is  about  100  MB  and  has  an 
unzipped-to-zip  ration  of  10:1. 

2.  Cloudsat-2B-Geoprof 

The  Cloudsat  2B-GeoProf  data  was  downloaded  from  the  Cloudsat  Data 
Processing  Center  and  are  in  a  Hierarchical  Data  Format  (HDF).  Each  file  has  37,100 
lines  of  data  and  the  name  convention  is:  the  Julian  date  and  granule  start  time  (hour, 
minute,  second),  granule  number,  type  of  sensor  (Cloudsat  =  CS),  data  product  type,  and 
the  suffix  ‘GRANULE_P_R04E03.hdf.’  For  example,  granule  19893  is  named 
2010023092101  _19893_CS_2B-GEOPROF  _GRANULE_  P_R04_E03.hdf.  One 
zipped  Cloudsat  data  file  is  about  13  MB  and  has  an  unzipped-to-zip  3:1. 

F.  METHODS 

The  Cloudsat  and  WWMCA  data  sets  are  on  different  temporal  and  spatial  grids 
that  have  different  resolutions,  and  differ  in  the  variables  they  use  to  describe  clouds.  So 
the  data  needed  to  be  processed  and  put  on  a  common  footing  before  it  could  be 
analyzed.  This  section  describes  the  programming  requirements  and  associated  tasks 
necessary  to  compare  and  analyze  the  two  sets  of  data  in  a  spreadsheet  program  (e.g., 
Microsoft  Excel).  The  major  challenge  was  getting  the  Cloudsat  data:  (a)  onto  the  same 
time  and  space  grid  as  the  WWMCA  data;  and  (b)  into  the  same  units  as  the  WWMCA 
data.  This  consisted  of  matching  Cloudsat  data  to  the  WWMCA  cells,  converting 

Cloudsat  cloud  mask  measurements  into  total  cloud  amount  percentages  for  a  given 
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WWMCA  cell,  and  binning  both  WWMCA  and  Cloudsat  data  into  common  cloud  cover 
events.  The  WWMCA  variable  of  interest  for  this  research  project  was  total  cloud 
amount. 

The  WWMCA  data  is  organized  into  128  square  boxes  (64  boxes  per  hemisphere) 
with  each  box  containing  128  cells  (Figure  8).  The  cells  have  unique  i-j  coordinates  that 
describe  the  latitude  and  longitude  of  the  cell.  The  cells  are  24  x  24  km,  true  at  60° 
latitude  (HQ  AFWA/DNXM  201 1). 


WWMCA  Box  and  Grid  Organization 


Northern  Hemisphere 


Boxes  1-8 
Boxes  9-16 


Boxes  57-154 


Southern  Hemisphere 


Boxes  101-103 
Boxes  109-116 


Boxes  157-164 


Figure  8.  The  WWMCA  grid  is  organized  into  128  square  boxes  (64  boxes  per 
hemisphere)  with  each  box  containing  128  cells.  The  cells  have  unique  i-j 
coordinates  that  describe  the  latitude  and  longitude  of  the  cell.  The  cells  are 
24  km  long  on  a  side  (true  at  60°  latitude).  The  yellow  shaded  boxes  are  the 
two  WWMCA  boxes  that  we  focused  on  in  this  study.  The  northern  box  is 
WWMCA  Box  29  and  the  southern  one  is  Box  22.  Base  figures  from  16th 

Weather  Squadron  (2011). 


Each  Cloudsat  granule  is  40,786  km  in  length  and  contains  37,088  profiles.  Each 

profile  represents  a  vertical  sounding  from  the  satellite  through  the  atmosphere  to  the 

surface.  The  horizontal  surface  area  represented  by  each  profile  is  the  Cloudsat  pixel 

area  (or  IFOV).  Each  Cloudsat  pixel  (Figure  5)  is  about  1.3  km  wide  (across  the  track) 

and  1.7  km  long  (along  the  track).  The  main  Cloudsat  data  of  interest  for  this  study  was 

the  Cloudsat  cloud  mask  data  in  the  2B-Geoprof  product,  which  provided  information 
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about  the  presence  or  absence  of  clouds,  along  with  corresponding  date-time  and  latitude- 
longitude  data.  This  data  was  available  for  each  profile  and  for  each  of  125  vertical 
levels  within  the  profile  (Figure  9).  The  cloud  mask  values  indicate  the  likelihood  of 
cloud  detection  and  are  described  in  Table  3.  The  Cloudsat  dataset  also  includes  a 
variable  that  describes  the  height  of  the  Cloudsat  levels  above  mean  sea  level  (MSL)  that 
we  used  to  filter  out  the  backscatter  from  the  surface  (see  next  section). 


Need  1  representative 
value  for  cloud  mask. 
This  representative 
value  will  be  the 
highest  cloud  mask 
value  in  the  column. 


Cloudsat  (altitude  of  705  km) 


Cloud  Profiling  Radar  (CPR)  Beam 


Top  of  Atmosphere 


Surface 


Column  1 


Vertical  CPR  levels 

(240  meter  resolution, 

124  levels  contained  in 
CPRCIoudmask  data) 

The  columns  listed  in  the  "CPR_Cloud_mask" 
parameter  of  the  granule  HDF  file  represent 
the  vertical  levels  of  the  CPR  starting  in  the 
upper  atmosphere  (column  1)  and  ending  at 
the  bottom  of  the  atmosphere  (column  124). 


Column  124 


Figure  9.  Vertical  levels  in  Cloudsat. 
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Figure  10  illustrates  the  processing  of  the  WWMCA  and  Cloudsat  data  that  we 
did  to  prepare  the  data  for  analysis,  as  well  as  the  analysis  of  the  processed  data. 


Figure  10.  Overview  of  data  processing  conducted  to  prepare  WWMCA  and  Cloudsat 

datasets  for  analysis. 

1.  Cloudsat  Data  Preparation 

Cloudsat  data  was  prepared  for  verifying  the  WWMCA  data  by  running  a  set  of 
Python  programs  that  extracted  the  required  data  from  a  data  base  containing  the  2010 
Cloudsat  data  files  that  intersected  WWMCA  Boxes  22  and  29.  The  required  data  were: 

•  profile  time 

•  UTC  start  time 

•  latitude 

•  longitude 

•  height 

•  CPR  cloud  masking 
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The  data  for  these  variables  were  extracted  and  matched  to  the  appropriate  WWMCA 
verification  time  and  cell.  The  Cloudsat  observations  were  matched  to  WWMCA 
verification  times  that  best  represented  the  atmospheric  cloud  state  for  that  time,  using  a 
similar  to  that  used  by  CDFS  II  to  match  satellite  observations  to  the  cloud  analysis. 
Cloudsat  observations  that  fall  within  the  first  (second)  half  of  an  hour  were  matched  to 
WWMCA  data  of  that  (the  following)  hour.  For  example,  if  a  Cloudsat  observation  was 
from  0817Z,  it  was  matched  to  the  0800  WWMCA  verification  time.  If  the  observation 
was  from  083 1Z,  it  was  matched  to  the  0900  verification  time.  Next,  the  Cloudsat  data 
was  mapped,  or  converted,  to  the  WWMCA  grid  and  matched  to  the  corresponding 
WWMCA  cells  within  the  WWMCA  box  of  interest,  either  Box  22  or  Box  29. 

We  used  the  height  variable  to  remove  from  the  data  processing  effort  any 
Cloudsat  observations  with  heights  less  than  1  km,  to  eliminate  backscatter  from  the 
ground.  We  also  identified  the  WWMCA  cells  that  contained  six  or  more  Cloudsat 
pixels,  so  that  we  could  focus  our  analyses  on  the  cells  and  times  for  which  there  was  at 
least  25%  coverage  by  Cloudsat  (six  Cloudsat  pixels  cover  25%  coverage  of  a 
WWMCA). 

Cloudsat  cloud  masking  data  are  available  at  125  vertical  levels  for  each  profile 
(Figure  9).  For  comparison  to  the  WWMCA  total  cloud  amount  data,  we  reduced  the  125 
values  to  a  single  representative  cloud  mask  value  for  each  profile.  We  used  the  highest 
cloud  mask  value  from  the  125  levels  as  the  representative  cloud  mask  value  (after 
deleting  the  values  from  the  lowest  1  km,  as  described  above).  There  are  pros  and  cons 
to  this  selection  of  the  cloud  mask  value.  In  particular,  this  selection  minimizes  the  risk 
of  overstating  the  cloud  masking,  but  it  may  underestimate  the  actual  cloud  masking. 
Recall  that  the  cloud  masking  values  represent  the  likelihood  of  occurrence  of 
hydrometeors  in  a  profile.  So  a  summation,  or  an  average,  of  values  from  more  than  one 
of  the  125  levels  might  well  overstate  the  likelihood  clouds.  Thus,  we  chose  to  use  the 
highest  cloud  mask  value  to  represent  the  full  profile. 
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We  assigned  a  cloud  occurrence  value  to  each  pixel  by  using  the  Cloudsat  cloud 
mask  value  for  each  Cloudsat  pixel  to  determine  if  cloud  or  no-cloud  conditions  had 
occurred.  If  the  cloud  mask  value  was  less  than  30,  we  assigned  a  cloud  occurrence  value 
of  0  to  indicate  a  no-cloud  condition.  If  the  cloud  mask  value  was  30  or  greater,  we 
assigned  a  cloud  occurrence  value  of  1  to  indicate  the  presence  of  cloud,  or  cloud 
conditions.  It  is  important  to  note  that  prior  studies  used  a  Cloudsat  cloud  mask  value  of 
20  as  the  cloud/no-cloud  threshold,  rather  than  the  value  of  30  that  we  used.  The 
Cloudsat  Standard  Data  Products  Handbook  (2008)  states  that  values  of  30  or  greater  are 
associated  with  the  greatest  confidence  in  cloud  detection.  Based  on  this,  we  decided  to 
use  a  cloud  mask  value  of  30  as  the  threshold  for  determining  whether  clouds  were  or 
were  not  present  in  a  profile.  However;  to  test  the  sensitivity  of  our  analysis  results  to 
this  threshold  selection,  we  also  used  a  threshold  value  of. 

Next,  we  converted  the  set  of  Cloudsat  cloud  and  no-cloud  condition  values  for  all 
the  pixels  in  a  WWMCA  cell  to  a  single  total  cloud  amount  for  the  cell  using  the 
following  calculation: 

Number  o  f  Cloudsat  cloudy  points  ...  ,  „ 

— - ; - : - r—. - ; - ; -  *  100  =  Total  cloud  amount  (%) 

Total  number  of  Cloudsat  points 

That  is,  we  divided  the  number  of  Cloudsat  pixels  that  indicated  the  presence  of  clouds 
within  a  WWMCA  cell  by  the  total  number  of  Cloudsat  pixels  in  the  cell,  and  then 
multiplied  by  100,  to  derive  a  total  cloud  amount  percentage  for  the  cell.  For  example,  if 
12  out  of  19  Cloudsat  pixels  in  a  WWMCA  cell  showed  cloud  conditions,  then  the  total 
cloud  amount  for  the  cell  was  calculated  to  be  63%  (Figure  1 1). 
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12  of  19  points  are  cloud.  The  total 
cloud  amount  is  63% 


Figure  11.  Schematic  illustration  of  the  overlap  of  a  Cloudsat  swath  (red  line)  and  a 
WWMCA  cell  (gray  box).  The  red  circles  indicate  the  Cloudsat 
representation  of  cloud  conditions  (solid  red  circles)  and  no-cloud  conditions 
(open  red  circles).  The  total  cloud  amount  for  the  cell,  as  indicated  by 
Cloudsat,  is  the  number  of  cloud  conditions  divided  by  the  number  of  cloud 
and  no-cloud  conditions,  all  multiplied  by  100.  For  this  case,  12  of  the  19 
circles  indicate  cloud  conditions,  so  the  total  cloud  amount  is  63%. 


The  resulting  Cloudsat  total  cloud  amount  data  was  then  binned  into  one  of  three  cloud 
cover  events  (Table  4).  Cells  that  had  a  total  cloud  amount  value  >  80%  were  assigned  to 
the  Definite-Cloud  event  bin.  Cells  that  had  a  total  cloud  amount  value  <  20%  were 
assigned  to  the  No-Cloud  event  bin.  Cells  with  in  between  total  cloud  amount  values 
were  assigned  to  the  Probable-Cloud  event  bin. 


Table  4.  Method  used  to  determine  cloud  cover  event  status  for  a  WWMCA  cell  based 

on  the  Cloudsat-based  total  cloud  amount  (%)  for  the  cell. 


Cloud  Event  Number 

Total  Cloud  Amount 

Event  Name 

1 

<  20  % 

No-Cloud 

2 

21—79  % 

Probable-Cloud 

3 

>  80  % 

Definite-Cloud 

2.  WWMCA  Data  Preparation 

We  needed  to  match  up  the  WWMCA  total  cloud  amount  data  to  the  Cloudsat 
data  in  the  CSV  file  that  was  generated  during  the  Cloudsat  data  preparation  stage  by 
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inserting  WWMCA  data  in  the  appropriate  rows.  The  rows  represent  the  time  and 
location  of  the  data.  So  inserting  the  WWMCA  data  into  the  correct  rows  meant  that  we 
did  a  temporal  and  spatial  match  of  the  Cloudsat  and  WWMCA  data. 

We  extracted  WWMCA  data  and  wrote  it  to  a  CSV  file  for  the  period  and 
location  of  the  Cloudsat  data  that  was  already  in  the  CSV  file.  However,  since  the 
WWMCA  data  has  a  lower  temporal  and  spatial  resolution  than  the  Cloudsat  data,  there 
are  rows  that  contain  Cloudsat  data  and  duplicate  WWMCA  data  from  adjacent  rows. 
All  of  the  WWMCA  information  contained  in  a  data  line  (Figure  3),  including  the  mean 
pixel  age  was  written  to  the  CSV  file  containing  the  Cloudsat  data. 

We  then  binned  WWMCA  total  cloud  amount  into  the  same  three  cloud  cover 
events  as  Cloudsat  total  cloud  amount  (Table  4). 

In  summary,  there  are  significant  differences  in  the  spatial  and  temporal 
resolutions  of  WWMCA  and  Cloudsat  data.  WWMCA  data  is  contained  in 
1024  x  1024  cells  per  hemisphere  with  a  horizontal  resolution  of  24  km.  Cloudsat  data 
has  a  1.3  x  1.7  km  footprint  and  is  referenced  to  latitude  and  longitudes.  CDFS  II  outputs 
a  new  WWMCA  product  every  hour,  and  Cloudsat  flies  over  a  WWMCA  cell  for  three 
minutes,  twice  daily.  This  indicates  three  significant  challenges  in  verifying  WWMCA 
against  Cloudsat  (and  many  other  observational  data  sets):  (1)  accurately  matching  the 
two  data  sets  in  space;  (2)  accurately  matching  the  two  data  sets  in  time;  and 
(3)  obtaining  a  sufficient  amount  of  independent  observational  data  to  use  in  verifying 
WWMCA  (finding  an  adequate  amount  of  independent  observations  that  fall  within  a 
WWMCA  cell  to  confidently  verify  the  WWMCA  analysis  for  that  cell).  Cloudsat  and 
many  other  global  observational  data  sets  are  readily  available  on  regular  latitude- 
longitude  grids.  Verification  of  WWMCA  would  be  a  much  more  straightforward 
process  if  WWMCA  data  was  also  readily  available  on  a  regular  latitude-longitude  grid. 

3.  Analyzing  Data  in  Microsoft  Excel 

Verification  of  the  WWMCA  cloud  analysis  to  Cloudsat  data  was  accomplished 
with  Microsoft  Excel.  Refer  to  Appendix  B  for  the  format  and  formulas  used  to  analyze 
the  data.  The  CSV  file  containing  the  processed  WWMCA  and  Cloudsat  data  for  a  given 
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month  was  imported  into  Excel  and  data  lines  with  values  of  “-1”  and  “less  than  6  points” 
were  filtered  out  of  the  analysis  process,  since  these  points  represented  missing  data  and 
WWMCA  cells  with  less  25%  area  covered  by  Cloudsat  observations,  respectively. 

Next  we  tallied  the  hits,  false  alarms,  misses,  and  correction  rejections  for  the 
three  cloud  cover  events  for  Cloudsat  cloud  thresholds  values  20  and  30.  The  tallies  were 
generated  for  use  in  entering  data  into  the  contingency  tables  we  used  to  verify  the 
performance  of  WWMCA.  Table  5  explains  the  logic  for  determining  whether  a  given 
WWMCA-Cloudsat  data  pair  would  be  counted  as  a  hit,  false  alarm,  miss,  and  correction 
rejection.  In  this  table,  the  upper  portion  describes  the  logic  for  verifying  the 
performance  of  WWMCA  in  analyzing  Definite-Cloud  events  (cloud  event  3),  the  middle 
portion  does  the  same  for  Probable-Cloud  events  (cloud  event  2),  and  the  bottom  portion 
does  the  same  for  No-Cloud  events  (cloud  event  1).  The  cell  letters  (A,  B,  C,  D)  refer  to 
the  four  cells  in  the  two  by  two  contingency  tables  that  we  used  (see  next  section  and 
Wilks  2006).  For  example,  the  third  row  of  Table  5  shows  that  if  WWMCA  analyzed 
cloud  event  3  and  Cloudsat  observed  cloud  event  3,  then  a  tally  would  be  entered  in  cell 
A  of  the  contingency  table,  which  is  the  Hit  cell. 


Table  5.  Logic  for  tallying  hits,  false  alarms  (False),  misses  (Miss)  and  correct 

rejections  (Reject)  when  comparing  a  WWMCA  cloud  cover  analysis  to  the 
corresponding  Cloudsat  cloud  cover  observation. 


Tallying  Logic  for  Verifying  WWMCA  Performance  in  Analyzing  Cloud  Cover  Events 

3 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

3 

3 

False 

B 

3 

lor  2 

Miss 

C 

lor  2 

3 

Reject 

D 

lor  2 

lor  2 

2 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

2 

2 

False 

B 

2 

1  or  3 

Miss 

C 

lor  3 

2 

Reject 

D 

lor  3 

1  or  3 

1 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

1 

1 

False 

B 

1 

2  or  3 

Miss 

C 

2  or  3 

1 

Reject 

D 

2  or  3 

2  or  3 
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Next,  we  took  the  tallied  categories  for  cloud  cover  events  and  then  flagged  the 
data  as  either  day  or  night.  This  was  done  by  using  the  monthly  mean  sunset  and  sunrise 
times  for  a  reference  point  within  a  WWMCA  box  that  we  assumed  to  be  representative 
to  the  entire  box  (Table  6).  The  reference  point  we  used  for  Box  22  was  Tehran,  Iran. 
The  reference  point  for  Box  29  was  Orenburg,  Russia.  In  Box  29,  data  lines  that  fell 
within  the  Arctic  Circle  (Cloudsat  latitudes  >  70°N)  were  flagged  as  either  day  or  night 
depending  on  the  time  of  year.  January  and  April  were  assumed  to  be  polar  night 
conditions,  and  July  and  October  were  assumed  to  be  polar  day  conditions.  We  then 
created  separate  tallies  for  the  day  and  night  hit,  false  alarm,  miss,  and  correction 
rejection  categories.  This  allowed  us  to  separately  assess  the  performance  of  WWMCA 
for  all  hours  of  the  day,  for  daylight  hours,  and  for  nighttime  hours. 
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Table  6. 


Mean  times  of  sunrise  and  sunsets  for  reference  points  within  WWMCA  Box  22  and  Box  29.  We  assumed  that  the 
reference  point  times  were  representative  of  the  entire  WWMCA  box.  The  daylight  times  were  used  to  flag  WWMCA 

and  Cloudsat  data  as  either  day  or  night  data. 


Box  22  Reference:  Tehran,  Iran  Daylight  Range 


Box  22 

Sunrise 

(Local) 

Sunset 

(Local) 

Sunrise 

(Local) 

Sunset 

(Local) 

GMT 

+6  hrs 

GMT 

+6  hrs 

Sunrise 

(Z) 

Sunset 

(Z) 

Sunrise 

(Z) 

Sunset 

(Z) 

Filter  Type 

Jan 

9:30 

17:30 

0.3958 

0.7292 

3:30 

0.1458 

0.5417 

0.8750 

13:00 

21:00 

Day 

Apr 

7:30 

21:00 

0.3125 

0.8750 

3:30 

0.1458 

0.4583 

1.0208 

11:00 

0:30 

Night 

Jul 

6:00 

22:30 

0.2500 

0.9375 

3:30 

0.1458 

0.3958 

1.0833 

9:30 

2:00 

Night 

Oct 

8:30 

19:30 

0.3542 

0.8125 

3:30 

0.1458 

0.5000 

0.9583 

12:00 

23:00 

Day 

Box  29  Reference:  Orenburg,  Russia  Daylight  Range 


Box  29 

Sunrise 

(Local) 

Sunset 

(Local) 

Sunrise 

(Local) 

Sunset 

(Local) 

GMT 

+6  hrs 

GMT 

+6  hrs 

Sunrise 

(Z) 

Sunset 

(Z) 

Sunrise 

(Z) 

Sunset 

(Z) 

Filter  Type 

Jan 

9:00 

18:00 

0.3750 

0.7500 

6:00 

0.2500 

0.6250 

1.0000 

15:00 

0:00 

Day 

Apr 

7:00 

21:00 

0.2917 

0.8750 

6:00 

0.2500 

0.5417 

1.1250 

13:00 

3:00 

Night 

Jul 

6:30 

22:30 

0.2708 

0.9375 

6:00 

0.2500 

0.5208 

1.1875 

12:30 

4:30 

Night 

Oct 

8:30 

19:30 

0.3542 

0.8125 

6:00 

0.2500 

0.6042 

1.0625 

14:30 

1:30 

Night 
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4. 


Contingency  Table  Analysis 


The  performance  of  WWMCA  was  verified  using  2x2  contingency  table 
methods  (Table  7;  Wilks  2006).  Even  though  WWMCA  is  an  analysis  we  are  treated  it 
as  an  hourly  forecast  so  that  we  could  assess  its  performance  using  standard  performance 
metrics.  We  calculated  a  number  of  performance  metrics  based  on  Table  7. 


Table  7.  The  format  of  the  2x2  contingency  table  analysis  used  to  verify  WWMCA 

cloud  analyses.  Based  on  Wilks  (2006). 


Observations 

(Cloudsat) 

Cloud 

No-cloud 

Analysis 

(WWMCA) 

Cloud 

A 

Hits 

(A/N) 

B 

False  Alarms 
(B/N) 

A  +  B 
(A  +  B)  /  N 

Marginal  Totals  for 
Analysis  & 
Marginal 
Distributions  for 
Analysis 

No-cloud 

C 

Misses 

(C/N) 

D 

Correct 

Rejections 

(D/N) 

C  +  D 

(C  +  D)/N 

A  +  C 
(A  +  C)  /  N 

B  +  D 
(B  +  D)  /  N 

N  =  A  +  B  +  C  +  D 
1.00 

Marginal  Totals  for  Observations 
&  Marginal  Distributions  for 
Observations 

Sample  Size  & 
Total  Probability 

We  calculated  a  number  of  WWMCA  performance  metrics  based  on  Table  7. 
The  probability  of  detection  (POD),  also  called  hit  rate  (H),  is  the  ratio  of  correct 
analyses  of  an  event  to  the  number  of  times  the  event  occurred.  In  our  case,  the  events  of 
interest  were  Definite-Cloud,  Probable-Cloud,  and  No-Cloud  events,  which  we  analyzed 
separately.  The  H  or  POD  was  calculated  by: 


Hits 

H  =  POD  =  - - - 

( Hits  +  Misses ) 


Proportion  correct  (PC),  which  Wilks  (2006)  states  is  the  most  straightforward 
and  sensitive  measure  of  the  accuracy  of  nonprobabilistic  forecasts  for  discrete  events, 
credits  correct  cloud  and  no-cloud  analyses  equally,  and  is  the  ratio  of  the  number  of 
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correct  analyses  to  the  total  number  of  samples.  However,  PC  does  not  distinguish 
between  correct  event  and  no-event  analyses.  We  calculated  the  proportion  of  occasions 
when  the  analysis  was  correct  by: 

(Hits  +  Correct  Rejections') 

Number  of  Samples 


We  also  calculated  the  threat  score  (TS),  or  critical  success  index  (CSI),  which 
eliminates  from  consideration  correct  rejections.  The  best  possible  threat  score  is  one  and 
the  worst  is  a  zero  (Wilks  2006)  and  was  calculated  by: 

Hits 

TS  =  CSI  =  - - - - - - - 

(Hits  +  False  Alarms  +  Misses) 


The  bias  (B)  is  the  total  number  of  analyses  of  an  event  divided  the  total  number 
of  observations  of  the  event.  Unbiased  analyses  would  have  a  bias  value  of  1.  Bias 
values  greater  than  one  indicate  that  the  event  occurred  more  often  than  it  was  observed 
(the  event  was  over  analyzed)  and  bias  values  less  than  one  indicate  the  event  occurred 
less  often  than  observed  (the  event  was  under  analyzed).  We  calculated  bias  by: 

Total  for  Analysis  ( Hits  +  False  Alarms) 

T otal  for  Observations  ( Hits  +  Misses) 

The  false  alarm  ratio  (FAR)  is  the  ratio  of  false  alarms  for  an  event  to  the  total 
number  of  analyses  of  the  event,  or  the  fraction  of  analyses  that  were  wrong  (Wilks 
2006).  Thus,  FAR  is  the  number  of  false  alarms  divided  by  the  sum  of  the  hits  and  false 
alarms,  all  for  the  event  of  interest.  The  best  value  for  FAR  is  zero  and  worst  value  is 
one.  The  false  alarm  ratio  was  calculated  by: 
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False  Alarms 


FAR  = 


T  otal  for  Analysis 


The  skill  of  the  analysis  was  tested  with  the  Heidke  skill  score  (HSS).  The 
Heidke  skill  score  is  based  on  the  portion  correct  as  the  basic  accuracy  measure.  Perfect 
analyses  would  yield  a  score  of  one;  a  score  of  zero  would  indicate  that  the  analyses  are 
equivalent  to  random  reference  analyses;  and  a  negative  score  would  indicate  that  the 
analyses  were  worse  than  random  analyses.  Wilks  (2006)  explains  that  the  reference 
measure  for  the  Heidke  skill  score  is  the  portion  that  is  correct  from  a  random  analysis 
statistically  independent  of  observations.  We  calculated  the  Heidke  skill  score  as: 


HSS 


2 {Hits  *  Correct  Rejections  —  False  Alarms  *  Misses) 

( Hits  +  Misses){Misses  +  Correct  Rejections )  +  (/fits  +  False  Alarms)( False  Alarms  +  Correct  Rejections) 


A  description  of  our  WWMCA  performance  results  is  provided  in  Chapter  IV. 


5.  Summary  of  Assumptions 

In  order  to  verify  WWMCA  against  Cloudsat,  we  made  a  number  of  important 
assumptions.  In  particular,  we  assumed  that: 


1.  Cloudsat  is  the  ‘truth’  for  describing  the  actual  clouds  in  the  atmosphere. 

2.  The  Cloudsat  level  with  the  high  cloud  mask  value  is  representative  of  all 
125  levels  and  provides  a  good  measure  of  the  total  cloud  mask. 

3.  Cloudsat  observations  that  cover  >25%  of  a  WWMCA  cell  are  adequate  to 
represent  all  the  clouds  in  the  cell  and  for  verifying  WWMCA  for  that  cell. 

4.  The  mean  monthly  sunrise/sunset  times  near  the  center  latitude  of  a 
WWMCA  box  are  representative  of  the  whole  box. 

5.  January,  April,  July,  and  October  are  representative  of  the  major  seasons. 

6.  Cloud  cover  can  be  adequately  represented  by  the  three  cloud  events 
described  in  Table  4. 
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7. 


WWMCA  performance  can  be  adequately  assessed  using  metrics  based  on 
2x2  contingency  tables. 


We  applied  these  assumptions  for  a  variety  of  reasons — in  particular  to  keep  our 
analyses  relatively  straightforward.  Assumption  1  is  not  easy  to  modify,  since  extensive 
independent  cloud  observations  are  scarce.  Assumptions  2-7  can  be  modified  and  we 
recommend  doing  so  in  future  studies.  In  addition,  we  recommend  testing  the  sensitivity 
of  the  performance  results  to  these  assumptions  (see  Chapter  IV). 
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III.  RESULTS 


A.  OVERVIEW 

In  this  chapter,  we  present  the  results  of  WWMCA  verification  for  WWMCA 
Boxes  22  and  29  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud 
threshold.  For  comparison,  WWMCA  verification  results  from  Cloudsat  cloud  mask 
value  of  20  for  cloud/no-cloud  threshold  are  shown  in  Appendix  C.  The  number  of 
occurrences  of  hits,  false  alarms,  misses,  and  correct  rejections  for  the  three  cloud  cover 
events — 3:  Definite-Cloud,  2:  Probable-Cloud,  and  1:  No-Cloud — described  in  section 
2.F.3,  were  used  to  characterize  WWMCA  performance  through  formulas  stated  in 
section  2.F.4.  The  performance  results  are  provided  for  each  of  the  three  cloud  cover 
events,  for  the  daytime  data,  for  nighttime  data,  and  for  combined  daytime  and  nighttime 
data  (referred  to  as  “All  Day”  results).  These  results  are  presented  for  each  of  the  four 
months  we  analyzed  (January,  April,  July,  and  October)  and  for  the  two  WWMCA  boxes 
we  analyzed  (22  and  29).  Note  that  in  the  figures  in  this  chapter,  Definite-Cloucl  events 
are  sometimes  referred  to  as  Cloud  events.  Our  emphasis  in  analyzing  WWMCA  was  on 
its  performance  in  analyzing  Definite-Cloud  and  No-Cloud  events,  since  these  are  of 
most  interest  to  IC  users  of  CDFS  II  products.  However,  we  also  present  in  the  following 
sections  the  results  for  WWMCA  performance  in  analyzing  Probable-Cloud  events, 
which,  as  shown  in  these  sections,  tends  to  be  quite  low. 

B.  WWMCA  BOX  22 

For  WWMCA  Box  22,  Figures  12  and  13  show  the  number  of  occurrences  of  hits 

(A),  false  alarms  (B),  misses  (C),  and  correct  rejections  (D)  for  each  cloud  cover  event, 

and  Figure  14  shows  the  marginal  distribution  of  the  cloud  events,  for  the  four  months. 

Figures  12  and  13  show  that  the  highest  number  of  occurrences  for  each  cloud  event  was 

in  the  correct  rejections  category.  No-Cloud  events  had  the  largest  number  of  hits  in  each 

month  and  Definite-Cloud  events  had  the  second  highest.  The  marginal  distributions  of 

cloud  events  for  both  WWMCA  and  Cloudsat  (Figure  14)  show  a  general  decline 

(increase)  in  Definite-Cloud  (No-Cloud)  events  from  January  through  October.  The 
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information  in  Figures  12-14  is  useful  in  assessing  the  number  and  percentage  of 
opportunities  that  WWMCA  had  to  correct  analyze  the  three  cloud  events  (Figure  14)  and 
how  well  WWMCA  did  in  analyzing  those  events  (Figures  12-14).  For  example,  Figure 
14  shows  that  in  July  and  October  there  were  high  percentages  of  No-Cloud  events,  but 
relatively  low  percentages  of  Definite-Cloud  events.  This  in  turn  indicates  that  in  July 
and  October:  (a)  No-Cloud  events  were  relatively  common  and  therefore  perhaps 
relatively  easy  to  correctly  analyze;  and  (b)  Definite-Cloud  events  were  relatively  rare 
and  therefore  perhaps  relatively  difficult  to  correctly  analyze. 


January  Daytime  Occurrences  WWMCA  (22)  -  Cloudsat  (30)  2010 


April  Daytime  Occurrences  WWMCA  (22)-  Cloudsat  (30)  2010 
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January  Nighttime  Occurrences  WWMCA  (22)  -  Cloudsat  (30)  2010 


April  Nighttime  Occurrences  WWMCA  (22)  -  Cloudsat  (30)  2010 
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Figure  12.  Number  of  2  x  2  contingency  table  occurrences  in  January  and  April  2010  for 
WWMCA  Box  22  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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Figure  13.  Number  of  2  x  2  contingency  table  occurrences  in  July  and  October  2010  for 
WWMCA  Box  22  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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Daytime  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(22)  -  Cloudsat  (30)  2010 
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Nighttime  Marginal  Distributions  of  Cloud  Cover  Events  for 
WWMCA  (22)  -  Cloudsat  (30)  2010 
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All  Day  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(22)  -  Cloudsat  (30)  2010 
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Figure  14.  Marginal  distributions  of  cloud  cover  events  for  WWMCA  and  Cloudsat  for 
WWMCA  Box  22:  Definite-Cloud,  Probable-Cloud,  and  No-Cloud).  The 
calculation  of  the  marginal  distributions  is  shown  in  Table  7.  The  top  panel 
shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data,  and 
the  bottom  panel  for  both  daytime  and  nighttime  data.  Results  based  on  using 
a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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The  POD  for  the  cloud  cover  events  are  shown  in  Figure  15.  Overall,  WWMCA 
did  the  best  for  No-Cloud  events  year  round — especially  during  the  daytime  and  in  July 
and  October,  when  No-Cloud  events  were  relatively  common  (Figure  14).  In  particular, 
WWMCA  analyses  of  No-Cloud  events  were  correct  85 — 95%  of  the  time.  But 
WWMCA  analyses  of  Definite-Cloud  events  were  correct  about  57 — 62%  of  the  time  for 
winter,  spring,  and  summer,  and  45%  of  the  time  in  the  fall.  The  mean  Daytime  No- 
Cloud  POD  was  95%  and  the  mean  nighttime  No-Cloud  POD  was  88%.  The  mean  POD 
of  Definite-Cloud  events  were  considerably  less:  daytime  ranged  36 — 64%  and  nighttime 
49 — 65%.  The  lowest  Definite-Cloud  POD  occurred  in  fall  daytime  and  summer 
nighttime.  The  POD  for  Definite-Cloud  events  are  similar  to  but  somewhat  lower  than 
those  from  UCAR  (2008),  which  reported  a  POD  for  Cloud  events  of  64%  (Chapter  I, 
Section  C).  The  relatively  high  (low)  POD  for  No-Cloud  (Definite-Cloud)  events  may  be 
related  to  the  abundance  (scarcity)  of  No-Cloud  (Definite-Cloud)  events,  especially  in 
July  and  October  (see  prior  discussion  of  Figure  14). 
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Daytime  Hit  Rate  WWMCA  (22)  -  Cloudsat  (30)  2010 


Figure  15. 
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WWMCA  probability  of  detection  (POD),  or  hit  rate  (H),  for  WWMCA  Box 
22  for  all  three  cloud  cover  events:  Definite-Cloud  (blue),  Probable-Cloud 
(red),  and  No-Cloud  (green).  The  top  panel  shows  the  results  for  daytime 
data,  the  middle  panel  for  nighttime  data,  and  the  bottom  panel  for  both 
daytime  and  nighttime  data.  Results  based  on  using  a  Cloudsat  cloud  mask 
value  of  30  for  the  cloud/no-cloud  threshold. 
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Figure  16  shows  the  WWMCA  FAR  for  Box  22,  which  range  from  5  to  40%. 
The  FAR  for  Definite-Cloud  was  relatively  low  in  winter  and  spring,  but  relatively  high 
in  summer  and  fall.  The  FAR  for  No-Cloud  events  showed  the  opposite  evolution,  from 
relatively  high  in  winter  and  spring  to  relatively  low  in  summer  and  fall.  This  is 
consistent  with  the  occurrence  and  marginal  distributions  results  (Figures  12-14),  with 
the  FAR  for  an  event  type  being  lower  when  the  event  type  is  common  and  higher  when 
the  event  type  is  rare.  The  FAR  results  also  correspond  with  the  POD  results,  especially 
the  low  FAR  for  No-Cloud  events  in  summer  and  fall  that  is  associated  with  a  high  POD 
of  95%.  The  results  shown  in  Figures  12-16  suggest  that  WWMCA  did  better  in 
persistent,  or  common,  cloud  cover  conditions — for  example,  better  in  analyzing 
Definite-Cloud  events  in  winter  and  spring  when  those  events  were  common,  and  better 
in  analyzing  No-Cloud  events  in  summer  and  fall  when  those  events  were  common. 
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Figure  16.  WWMCA  false  alarm  ratios  for  WWMCA  Box  22  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 


The  WWMCA  PC  results  for  Box  22  are  shown  in  Figure  17.  The  three  cloud 
cover  events  had  similar  performance,  with  better  performance  in  the  summer  and  fall 
than  winter  and  spring.  Day  and  nighttime  PC  were  within  approximately  15%  of  each 
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other  throughout  the  year.  Overall,  the  PC  results  for  Definite-Cloud  events  were  a  little 
better  than  for  No-Cloud  events.  The  PC  results  are  consistent  with  the  POD  and  FAR 
results  in  indicating  that  WWMCA  did  well  in  persistent  conditions. 
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Figure  17.  WWMCA  proportion  correct  (PC)  for  WWMCA  Box  22  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 
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The  PC  performance  metrics  gives  equal  credit  for  hits  and  correct  rejections 
(Chapter  II,  Section  F.4;  Wilks  2006).  Thus,  we  also  calculated  the  threat  score  (TS)  to 
focus  the  performance  analysis  on  the  hits.  The  TS  results  for  Box  22  are  shown  in 
Figure  18  and  reveal  uniformly  better  No-Cloud  performance  than  Definite-Cloud 
performance,  with  the  best  performance  being  for  No-Cloud  in  the  summer  and  fall 
(about  85%).  Daytime  TS  for  no-cloud  was  about  68%  in  winter  and  spring  and 
increased  to  about  85%  in  summer  and  fall;  TS  for  definite  cloud  was  about  50%  in 
winter  and  spring  and  decreased  to  45%  in  summer  and  34%  in  fall.  Nighttime  TS  for 
no-cloud  was  about  65%  in  winter  and  spring  and  increased  to  87%  in  summer  and  fall; 
definite  cloud  was  52%  in  winter  and  spring  and  increased  to  about  42%  in  summer  and 
fall.  The  high  TS  for  No-Cloud  during  summer  and  fall  is  consistent  with  the  POD  of 
95%  (Figure  15)  and  the  low  FAR  of  10%  (Figure  16).  The  higher  TS  for  Definite-Cloud 
events  in  winter  and  spring  are  related  to  Definite-Cloud  events  having  been  relatively 
common  in  those  months  (Figure  14).  Taken  together,  these  results  support  the  idea  that 
WWMCA  performs: 

1.  better  when  cloud  conditions  that  are  relatively  persistent  and/or  common 

2.  worse  when  analyzing  cloud  conditions  that  are  relatively  variable 
(spatially,  temporally)  and/or  uncommon. 
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Figure  18.  WWMCA  threat  scores  (TS)  for  WWMCA  Box  22  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  19  shows  the  bias  (B)  and  indicates  that  No-Cloud  events  were  over 
analyzed  in  winter,  spring,  and  fall,  and  were  mostly  unbiased  in  summer.  Definite- 
Cloud  events  were  consistently  under  analyzed,  especially  in  the  fall,  with  the  exception 
of  summer  when  the  analysis  of  Definite-Cloud  events  was  mostly  unbiased.  These 
results  are  consistent  with  the  findings  of  the  UCAR  (2008)  and  AER  (2010)  studies. 
There  is  a  larger  bias  for  daytime  Definite-Cloud  and  No-Cloud  events  in  winter  and  fall, 
as  opposed  to  spring  and  summer,  and  there  is  a  generally  lower  bias  for  Definite -Cloud 
events  at  night.  These  results  are  also  consistent  with  the  results  shown  in  Figures  15-18 
For  example,  the  over  (under)  analysis  of  No-Cloud  (Definite-Cloud)  in  October  is 
consistent  with  the  high  (low)  POD  in  that  month. 
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Figure  19.  WWMCA  bias  (B)  for  WWMCA  Box  22  for  all  three  cloud  cover  events: 

Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green).  The  top 
panel  shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data, 
and  the  bottom  panel  for  both  daytime  and  nighttime  data.  The  heavy  dark 
line  at  B  =  1  indicates  an  unbiased  analysis.  Values  greater  (less)  than  1 
indicate  a  cloud  cover  event  that  was  over  (under)  analyzed.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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The  Heidke  skill  scores  are  shown  in  Figure  20  and  show  relatively  little 
difference  between  the  months  or  between  day  and  night.  The  highest  skill  scores  were 
associated  with  No-Cloud,  ranging  from  about  50%  in  January  night  to  65%  in  October 
night.  The  HSS  results  for  Definite-Cloud  events  are  similar  to  but  somewhat  less  than 
those  for  No-Cloud  events.  As  with  the  other  performance  metrics,  the  poorest 
performance  for  Definite-Cloud  events  is  in  October.  The  HSS  for  Probable-Cloud 
events  are  considerably  lower  than  for  the  other  two  events,  ranging  from  8%  to  13% 
throughout  the  year.  The  HSS  results  are  consistent  with  the  POD,  FAR,  PC,  and  TS 
results.  However,  it  is  interesting  to  note  that  HSS  shows  relatively  small  differences  in 
July  and  October  between  No-Cloud  and  Definite-Cloud,  unlike  the  case,  in  general,  for 
POD,  FAR,  and  TS.  This  may  be  because  HSS  gives  more  (less)  credit  for  good 
performance  in  analyzing  uncommon  (common)  events — and  in  July  and  October; 
Definite-Cloud  (No-Cloud)  events  were  uncommon  (common)  (see  Figure  14). 
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Figure  20.  WWMCA  Heidke  skill  score  (HSS)  for  WWMCA  Box  22  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 

The  heavy  dark  line  at  the  zero  value  indicates  that  the  WWMCA  is 
equivalent  to  reference  analyses  (random  analyses  statistically  independent  of 
observations).  A  perfect  WWMCA  cloud  analysis  would  have  a  value  of  one. 
Negative  values  indicate  WWMCA  performs  worse  than  reference  analyses. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 


49 


c. 


WWMCA  BOX  29 


For  Box  29,  we  analyzed  points  north  of  70°  N  as  being  in  polar  night  during  all 
of  January  and  all  of  April,  and  in  polar  day  during  all  of  July  and  all  of  October.  This  is 
an  incorrect  way  to  analyze  the  April  and  October  data  that  arose  due  to  an  error  in  the 
analysis  coding.  Thus,  for  the  April  and  October  results,  readers  should  focus  their 
attention  on  just  the  All  Day  results. 

Figures  21  and  22  show  the  number  of  occurrences  of  hits  (A),  false  alarms  (B), 
misses  (C),  and  correct  rejections  (D)  for  each  cloud  cover  event,  and  Figure  23  shows 
the  marginal  distribution  of  the  cloud  events,  for  the  four  months.  Note  that  there  was  a 
more  even  distribution  of  the  three  cloud  event  types  for  Box  29  (Figures  21  and  22)  than 
for  Box  22  (Figures  12  and  13).  In  particular,  the  marginal  distribution  of  cloud  events 
for  WWMCA  and  Cloudsat  (Figure  23)  show  that  neither  Definite-cloud  nor  No-Cloud 
events  dominated  in  2010  in  Box  29  as  opposed  to  Box  22.  Thus,  Box  29  in  2010 
represented  more  mixed  and  variable  cloud  conditions  than  did  Box  22  in  2010.  Box  29 
for  2010  may  also  have  presented  WWMCA  with  a  more  challenging  analysis  problem 
than  did  Box  22,  since  Box  29  showed  less  persistence  of  any  one  cloud  event  type.  Note 
too  that  WWMCA  understated  the  percentage  of  Definite-Cloud  events,  and  overstated 
the  number  of  No-Cloud  events,  except  during  July  (Figure  23).  WWMCA  also 
overstated  the  percentage  of  Probable-Cloud  events  in  all  months. 
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Figure  21 .  Number  of  2  x  2  contingency  table  occurrences  in  January  and  April  2010  for 
WWMCA  Box  29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 


51 


Numberof  Occurrences  Numberof  Occurrences  Numberof  Occurrences 


July  Daytime  Occurrences  WWMCA  (29)  -  Cloudsat  (30)  2010 


October  Daytime  Occurrences  WWMCA  (29)  -  Cloudsat  (30)  2010 


A  B  C  D 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


250000 


A  B  C  D 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


A  B  C  D 


October  Nighttime  Occurrences  WWMCA  (29)  -  Cloudsat  (30)  2010 


July  All  Day  Occurrences  WWMCA  (29)  -  Cloudsat  (30)  2010 


12  3  4 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


October  All  Day  Occurrences  WWMCA  (29)  -  Cloudsat  (30)  2010 


350000 

300000 

8 

=  250000 

£ 

|  200000 
2  150000 

Ji 

E  100000 

s 

z 

50000 

0 


A  B  C  D 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


Figure  22.  Number  of  2  x  2  contingency  table  occurrences  in  July  and  October  2010  for 
WWMCA  Box  29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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Daytime  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(29)  -  Cloudsat  (30)  2010 
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All  Day  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(29)  -  Cloudsat  (30)  2010 
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Figure  23.  Marginal  distributions  of  cloud  cover  events  for  WWMCA  and  Cloudsat  for 
WWMCA  Box  29:  Definite-Cloud,  Probable-Cloud,  and  No-Cloud).  The 
calculation  of  the  marginal  distributions  is  shown  in  Table  7.  The  top  panel 
shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data,  and 
the  bottom  panel  for  both  daytime  and  nighttime  data.  Results  based  on  using 
a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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The  POD  for  the  cloud  cover  events  are  shown  in  Figure  24.  Note  that  the  All 
Day  results  show  PODs  for:  (1)  No-Cloud  events  were  71-78%  in  winter  and  spring,  but 
51-60%  in  summer  and  fall;  (2)  Definite-Cloud  events  were  42%  in  winter  and  spring, 
64%  in  summer,  and  41%  in  fall;  and  (3)  Probable-Cloud  events  were  15%-29%,  with  an 
increasing  trend  throughout  the  year.  These  POD  results  were  considerably  lower  than 
those  for  Box  22  (Figure  15).  The  Definite-Cloud  results  are  also  lower  than  the  64% 
POD  for  cloud  reported  in  UCAR  (2008). 
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Figure  24.  WWMCA  probability  of  detection  (POD),  or  hit  rate  (FI),  for  WWMCA  Box 
29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue),  Probable-Cloud 
(red),  and  No-Cloud  (green).  The  top  panel  shows  the  results  for  daytime 
data,  the  middle  panel  for  nighttime  data,  and  the  bottom  panel  for  both 
daytime  and  nighttime  data.  Results  based  on  using  a  Cloudsat  cloud  mask 
value  of  30  for  the  cloud/no-cloud  threshold. 
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Figure  25  show  that  Definite-Cloud  events  had  an  All  Day  FAR  of  about  22%  to 
34%  throughout  the  year,  except  for  summer  when  the  FAR  was  41%.  This  summer 
increase  in  FAR  is  consistent  with  Figure  23,  which  shows  the  marginal  distribution  of 
WWMCA  exceeded  the  marginal  distribution  of  Cloudsat  in  July.  The  All  Day  FAR  for 
No-Cloud  events  ranged  from  33%  to  46%  throughout  the  year,  except  in  October  when 
it  increased  to  69%.  Note  that  Figure  23  shows  that  the  WWMCA  No-Cloud  marginal 
distribution  was  double  that  of  Cloudsat  in  October.  Note  that  the  FAR  values  for  Box 
29  were  considerably  higher  than  for  Box  22  (Figure  16). 
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Figure  25.  WWMCA  false  alarm  ratios  for  WWMCA  Box  29  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 


55 


The  WWMCA  PC  results  for  Box  29  are  shown  in  Figure  26.  The  All  Day 
Definite-Cloud  and  No-Cloud  events  had  similar  scores,  about  64%,  throughout  the  year, 
although  Definite-Cloud  decreased  to  53%  in  fall.  The  lower  PC  for  Definite-Cloud  in 
October  is  consistent  with  the  lower  Definite-Cloud  POD  values  in  October  (Figure  24). 
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Figure  26.  WWMCA  proportion  correct  (PC)  for  WWMCA  Box  29  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cioudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 
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The  TS  results  for  Box  29  are  shown  in  Figure  27  and  reveal  similar  No-Cloud 
and  Definite-Cloud  performance.  All  Day  No-Cloud  TS  was  slightly  better  in  winter 
spring  and  All  Day  Definite-Cloud  TS  was  slightly  better  in  summer  and  fall.  Box  29 
had  considerably  lower  TS  scores  than  Box  22  (Figure  18),  which  is  consistent  with  the 
lower  POD  (Figure  24)  and  higher  FAR  (Figure  25)  for  Box  29.  These  results  suggest 
that  WWMCA  performance  is  worse  when  there  is  relatively  high  variability  and  low 
persistence  in  the  cloud  event  types  (Figures  21-27  for  Box  29)  and  better  when  there  is 
relatively  low  variability  or  high  persistence  (Figures  12-20  for  Box  22). 
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Figure  27.  WWMCA  threat  scores  (TS)  for  WWMCA  Box  29  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 


Figure  28  shows  the  bias  (B)  results  for  Box  29.  Definite-Cloud  (No-Cloud) 
events  were  substantially  under-analyzed  (over  analyzed)  in  winter,  spring  and  fall, 
consistent  with  the  marginal  distribution  results  (Figure  23).  The  four  month  average  All 
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Day  Definite-Cloud  (No-Cloud)  bias  was  approximately  1.5  (-0.6).  The  overall  patterns 
in  Figure  28  are  similar  to  those  for  Box  22  (Figure  19)  but  with  larger  bias  magnitudes 
than  for  Box  22. 


Figure  28. 
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WWMCA  bias  (B)  for  WWMCA  Box  29  for  all  three  cloud  cover  events: 
Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green).  The  top 
panel  shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data, 
and  the  bottom  panel  for  both  daytime  and  nighttime  data.  The  heavy  dark 
line  at  B  =  1  indicates  an  unbiased  analysis.  Values  greater  (less)  than  1 
indicate  a  cloud  cover  event  that  was  over  (under)  analyzed.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no-cloud  threshold. 
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The  Heidke  skill  scores  for  Box  29  are  shown  in  Figure  29  and  are  similar  for  day 
and  night,  and  for  all  months.  The  HSS  values  are  in  the  1 1%  to  32%  range  for  Definite- 
Cloud  and  No-Cloud,  and  -4%  to  9%  range  for  Probable-Cloud.  These  relatively  low 
HSS  values,  compared  to  Box  22,  are  consistent  with  the  lower  POD,  PC,  TS  and  the 
higher  FAR  for  Box  29  than  for  Box  22.  Note  that  the  Box  29  HSS  values  show 
relatively  small  differences  through  the  year  between  No-Cloud  and  Definite-Cloud 
events,  unlike  the  case,  in  general,  for  POD,  FAR,  and  TS.  Similar  to  Box  22,  this  may 
be  because  HSS  gives  more  (less)  credit  for  good  performance  in  analyzing  uncommon 
(common)  events  (Figure  23). 

Positive  HSS  indicates  skill  above  a  random  analysis  (see  Chapter  II,  Section 
F.4).  However,  HSS  values  that  don’t  exceed  32%  for  Box  29  (Figure  29)  or  65%  for 
Box  22  (Figure  20)  are  problematic,  given  that  WWMCA  is  a  near  real-time  analysis  (as 
opposed  to  a  non-zero  lead  forecast)  that  is  based  on  many  real-time  observational  data 
sources.  Unfortunately,  there  do  not  appear  to  be  any  other  studies  of  WWMCA  that 
have  reported  HSS  results  or  other  skill  scores  that  compare  WWMCA  to  a  reference,  or 
benchmark,  analysis. 
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Figure  29.  WWMCA  Heidke  skill  score  (HSS)  for  WWMCA  Box  29  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 

The  heavy  dark  line  at  the  zero  value  indicates  that  the  WWMCA  is 
equivalent  to  reference  analyses  (random  analyses  statistically  independent  of 
observations).  A  perfect  WWMCA  cloud  analysis  would  have  a  value  of  one. 
Negative  values  indicate  WWMCA  performs  worse  than  reference  analyses. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  30  for  the  cloud/no¬ 
cloud  threshold. 


61 


D.  SUMMARY  OF  RESULTS 


The  results  show  that  WWMCA  performance  was:  (1)  better  for  the  southwest 
Asia  region  where  there  was  more  persistence  in  cloud  cover  events  (especially  No-Cloud 
events);  and  (2)  worse  for  the  western  Russia  -  Barents  Sea  region  where  there  was  more 
variability  in  cloud  cover  events.  This  makes  some  sense,  because  persistent  conditions 
tend  to  be  easier  to  analyze.  The  marginal  distribution  and  HSS  results  provide  a  good 
summary  of  how  differences  in  performance  were  associated  with  differences  in  cloud 
cover  event  persistence.  The  relatively  good  performance  for  the  southwest  Asia  region 
may  be  due  to  the  relatively  clear  skies  that  are  typical  of  that  region,  and  that 
characterized  the  region  during  much  of  2010,  especially  in  summer  and  fall.  Another 
factor  that  may  have  led  to  the  relatively  low  performance  for  the  western  Russia  - 
Barents  Sea  region  may  be  the  higher  pixel  ages  for  the  higher  latitudes  in  which  polar 
orbiting  satellites  are  a  main  source  of  cloud  information  for  WWMCA  (see  Chapters  II 
and  IV  for  more  on  this  topic). 

There  were  also  notable  differences  in  the  results  for  the  different  months,  and  for 
day  and  night  (e.g.,  in  the  All  Day  POD  and  HSS  results  for  Box  29,  and  in  the  Box  22 
bias  results  for  day  and  night). 

Overall,  the  WWMCA  performance  results  appear  to  be  at  least  somewhat 
problematic,  since:  (1)  WWMCA  products  are  near  real-time  analyses  based  on  many 
near  real-time  observational  data  sources;  and  (2)  WWMCA  is  used  to  initialize  cloud 
forecast  models  that  are  used  in  IC  planning. 
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IV.  SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

We  conducted  a  verification  of  the  WWMCA  cloud  analysis  product  generated  by 
CDFS  II  that  is  used  in  the  planning  of  overhead  imagery  collection.  Our  independent 
verification  data  source  was  data  from  the  Cloudsat  atmospheric  sounder,  a  satellite- 
borne  radar  system  designed  to  sense  condensed  cloud  particles.  We  looked  at  the  cloud 
analyses  of  two  WWMCA  boxes  that  are  of  high  interest  to  the  national  intelligence 
community  and  that  differ  in  their  meteorological  characteristics.  One  box  was  centered 
in  western  Russia  and  the  nearby  polar  region,  and  the  other  box  was  in  the  mid-latitudes 
and  contained  mountain,  desert,  ocean,  and  coastal  regions.  We  analyzed  one  year  of 
data  for  these  two  WWMCA  boxes  to  test  a  system  for  routinely  monitoring  WWMCA 
performance. 

Our  goal  was  to  analyze  the  performance  of  WWMCA  for  a  range  of  cloud  cover 
events,  times,  and  locations,  and  using  a  range  of  performance  metrics.  To  do  so,  we  had 
to  develop  a  method  to  match  the  data  sets  and  to  ensure  an  adequate  amount  of  ground 
truth  measurements  fell  within  each  WWMCA  cell  to  confidently  verify  that  cell’s  cloud 
analysis.  The  WWMCA  data  set  we  used  was  provided  by  the  16th  Weather  Squadron. 
The  Cloudsat  data  was  downloaded  from  the  Cloudsat  Data  Processing  Center  website. 
After  overcoming  the  differences  in  temporal  and  spatial  resolutions  of  the  data,  we  were 
able  to  calculate  metrics  that  provide  useful  information  about  the  performance  of 
WWMCA. 

B.  CONCLUSIONS 

Our  performance  metrics  showed  that  WWMCA  performed  better  under 
persistent  cloud  cover  situations  than  more  variable  cloud  situations.  For  instance, 
WWMCA  had  a  high  POD  for  No-Cloud  events  when  there  were  persistently  few  clouds. 
For  mid-  and  high  latitudes,  Definite-Cloud  events  were  under  analyzed  and  No-Cloud 
events  were  over  analyzed,  except  for  the  summer  when  both  events  were  unbiased.  The 
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Heidke  skill  scores  for  mid-latitude  cloud  analyses  were  considerably  higher  than  those 
for  high  latitude  cloud  analyses — for  example,  high  HSS  values  in  the  mid-latitudes  of 
about  60%  versus  about  30%  for  the  high  latitudes.  Performance  in  analyzing  Probable- 
Cloud  events  was  especially  poor  in  both  regions.  For  example:  (1)  HSS  for  Probable- 
Cloud  events  ranged  between  -5%  and  15%  in  both  mid-  and  high-latitudes;  and  (2)  the 
Heidke  Skill  Scores  were  less  than  14%  for  Boxes  22  and  29  year  round. 

The  better  performance  of  WWMCA  in  the  mid-latitudes  could  be  accounted  for 
by  the  higher  temporal  resolution  of  the  geostationary  satellites  that  provide  the  majority 
of  the  observations  at  those  latitudes  (evident  in  the  pixel  ages  listed  in  Table  8).  The 
mean  pixel  ages  for  Box  22  are  34-36  minutes  and  have  a  maximum  standard  deviation 
of  11  minutes.  Conversely,  the  low  temporal  resolution  of  observational  imagery  from 
polar-orbiting  satellites  could  explain  why  the  high  latitude  cloud  analyses  had  lower 
skill.  The  mean  pixel  ages  for  Box  29  are  on  the  order  of  three  hours  and  standard 
deviations  are  about  two  hours  (Table  8). 


Table  8.  Pixel  age  information  for  satellite  observations  used  in  the  WWMCA 
development  process.  Means  and  standard  deviations  calculated  from  the 
pixel  age  data  in  the  WWMCA  data  set.  Negative  minutes  are  due  to  a 
portion  of  the  satellite  imagery  coming  in  after  the  :00/:30  time  hack  (Connor 

2012,  personal  communication). 
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-3 

497 
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We  determined  that  WWMCA  performance  is  relatively  insensitive  to  reasonable 
choices  of  the  Cloudsat  cloud  masking  threshold  (values  between  20  and  30),  as  shown 
by  a  comparison  of  our  results  in  Chapter  III  to  those  in  Appendix  C. 

The  WWMCA  performance  results  that  we  developed  are  generally  lower  than  to 
those  reported  in  the  available  documentation  for  prior  studies  (e.g.,  UCAR  (2008). 
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However,  reports  of  performance  results  from  prior  studies  are  sparse  (see  Chapter  I, 
Section  C).  In  addition,  we  verified  WWMCA  performance  for  larger  areas  and  a  wider 
range  of  months  than  most  prior  studies,  and  we  verified  WWMCA  using  a  wider  range 
of  metrics.  The  paper  that  summarized  most  of  the  findings  of  the  prior  studies,  NGIS 
(2011),  disclosed  only  the  POD  and  not  the  FAR  or  other  scores  that  might  have  been 
generated  in  those  studies.  Of  course,  determining  performance  based  only  on  POD  or 
any  other  single  metrics,  is  very  problematic  (cf.  Wilks  2006). 

C.  RECOMMENDATIONS 

Future  analyses  of  WWMCA  results  and  its  operational  impacts  on  imagery 
collection  are  necessary  to  truly  assess  its  performance.  We  recommend  that  future 
studies  expand  on  the  regions,  months,  and  years  examined  to  evaluate  the  regional-scale, 
global-scale,  and  long-term  performance  of  WWMCA.  The  analysis  of  many  years  of 
data  would  help  address  issues  raised  by  unusual  events,  such  as  the  record  high 
temperatures  and  extensive  wildfires  in  western  Russia  in  summer  2010  (which  may  have 
had  an  impact  on  both  Cloudsat  observations  and  WWMCA  analyses  in  2010). 

Our  results  raised  the  question:  Is  WWMCA  more  accurate  when  pixels  ages  are 
small?  We  recommend  an  analysis  that  attempts  to  answer  that  question  by  testing  the 
sensitivity  of  WWMCA  performance  to  pixel  age,  especially  the  performance  at  high 
latitudes. 

A  future  study  should  test  the  sensitivity  of  our  assumptions,  as  discussed  at  the 
end  of  Chapter  II.  For  example,  we  recommend  a  future  study  to  test  the  minimum 
Cloudsat  pixels  necessary  to  confidently  represent  a  WWMCA  cell.  Recall  in  Chapter  II 
that  we  assumed  that  a  minimum  of  six  Cloudsat  pixels  (25%  coverage  of  a  WWMCA 
cell)  could  adequately  represent  a  WWMCA  cell. 

Future  studies  should  use  a  more  dynamic  method  for  determining  sunrise/  and 
sunset  times  method  to  flag  the  data  as  either  day  or  night.  Recall  in  Chapter  II  that  we 
chose  the  mean  monthly  sunrise  and  sunset  times  or  a  point  close  to  the  center  of  a 
WWMCA  box  (Table  6)  to  represent  that  entire  WWMCA  box,  and  that  for  the  polar 
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region  (>  70°  N),  we  flagged  data  as  either  polar  night  or  polar  day  based  on  the  season. 
These  approximations  were  used  to  reduce  the  data  processing  time  but  also  introduced 
some  errors  in  assigning  the  time  of  day  to  the  data  that  should  be  removed  in  future 
studies  that  focus  on  distinguishing  day  and  night  performance.  Applying  a 
sunrise/sunset  library  to  the  data  matching  stage,  such  as  the  one  used  in  Selin  (2011, 
personal  communication)  study,  would  help  remove  these  errors. 

We  found  that  WWMCA  performs  better  in  persistent  cloud  cover  situations — as 
indicated,  for  example,  by  our  HSS  results.  We  recommended  that  a  future  study 
compare  the  skill  of  WWMCA  to  that  of  persistent  analysis  to  test  the  validity  of  this 
finding  and  to  better  assess  WWMCA  skill. 

Additionally,  we  recommend  an  in-depth  study  that  will  exploit  the  methods 
developed  in  our  study  to  analyze  large  amounts  of  WWMCA  data  and  show  the  long¬ 
term  performance  of  WWMCA.  One  benefit  of  such  a  study  is  that  it  would  help 
determine  the  value  added  by  past  and  future  changes  to  CDFS  n.  This  study  should 
look  at  long  time  periods,  various  WWMCA  boxes  from  both  hemispheres,  and  a  range 
of  climate  conditions  (e.g.,  high  and  low  cloud  tops;  high  and  low  cloud  variability;  high 
and  low  cloud  amounts;  land,  ocean,  snow,  ice,  mountain,  and  desert  surfaces;  tropical, 
mid-latitude,  and  polar  regions;  etc.)  to  measure  the  overall  and  specific  aspects  of 
WWMCA  performance. 

The  operational  impacts  of  cloud  analyses  and  forecasts  are  being  tested  in  an 
ongoing  study  at  NPS  by  Lieutenant  Brian  Moore  (classified  thesis  expected  to  be 
published  in  September  2012,  Figure  1).  We  recommend  a  review  of  his  study  when 
completed  to  gain  insight  into  the  operational  significance  of  WWMCA  in  imagery 
collection. 

For  access  to  the  data  sets  and  computer  coding  used  in  this  study,  contact  Dr. 
Tom  Murphree  in  the  Department  of  Meteorology  at  the  Naval  Postgraduate  School 
(murphree@nps.edu). 


"Who  is  wise  enough  to  number  all  the  clouds...?" 
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APPENDIX  A.  CLOUD  DEPICTION  FORECAST  SYSTEM  (CDFS) 

II  PROCESSING  LEVELS 


Satellite  and  conventional  observations  (surface  observations  and  upper  air 
soundings)  undergo  a  four  level  process  to  be  merged  into  a  global  cloud  analysis  (Figure 
30).  Level  one  is  data  calibration,  level  two  classifies  each  pixel  into  cloudy  or  clear, 
level  three  applies  cloud  layering  and  typing,  and  level  four  consists  of  merging  the 
separate  analyses  into  one  global  analysis  (HQ  AFWA/DNXM  2011). 


Figure  30.  Illustration  of  the  four  processing  levels  within  Cloud  Depiction  Forecast 
System  II.  Observations  are  received  from  meteorological  satellites, 
conventional  observations,  and  global  analysis  from  various  models.  These 
observations  are  merged  into  one  global  cloud  analysis  that  is  used  to  initiate 
cloud  forecast  models.  Shapes  are  defined  as:  rectangles  are  processes; 
rounded  rectangles  are  inputs;  ovals  are  products;  and  snipe  same  side  corner 

rectangles  are  cloud  models. 
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A. 


LEVEL  1 


Level  one  processing  consists  of  data  ingestion  and  calibration.  Satellite 
telemetry  transmissions  are  received  by  AFWA’s  Satellite  Data  Handling  System  located 
at  Offutt  Air  Force  Base  in  Nebraska.  Downlinked  satellite  data  is  encoded  and  must  be 
decoded.  The  decoded  data  reveal  physical  parameters  for  radiance  measurements 
received  by  the  detectors  on  the  sensor’s  focal  plane  array  and  are  placed  in  the  Sensor 
Data  Records  (SDR).  Satellite  imagers’  focal  plane  array  consist  of  detectors  that 
represent  the  pixels,  which  measure  the  radiance  received  from  reflected  or  emitted 
energy  within  Earth’s  atmosphere.  CDFS  II  use  the  data  from  the  satellite’s  visible  and 
infrared  channels.  A  calibration  step  for  the  infrared  data  converts  the  measured  emitted 
energy  into  either  radiance  or  brightness  temperature.  Brightness  temperature  is  the 
temperature  of  an  object  if  it  was  radiating  as  a  black  body.  The  brightness  temperature 
is  the  parameter  required  by  CDFS  II  to  make  the  analysis.  Reflectance  values  are 
measured  from  the  satellite’s  visible  channels  are  used  directly  in  the  algorithms  (HQ 
AFWA/DNXM  2011). 

B.  LEVEL  2 

Level  two  is  where  cloud  detection  occurs.  Each  sensor  has  its  own  tailored 
algorithms  designed  to  optimize  their  instrument’s  ability  to  exploit  measurements  made 
in  different  channels  in  an  attempt  to  distinguish  cloud  from  clear  scene.  DMSP’s 
Operational  Line  Scanner  (OLS)  sensor,  has  the  highest  spatial  resolution,  but  only  two 
broadband  channels  (one  visible  and  one  infrared),  whereas,  NOAA  POES’  Advanced 
Very-High  Resolution  Radiometer  (AVHRR),  have  six  narrowband  channels  (one 
visible,  one  near-infrared,  and  four  infrared).  DMSP’s  OLS  cloud  detection  is 
accomplished  by  comparing  a  pixel’s  brightness  temperature  to  a  cloud-free  referenced 
pixel’s  brightness  temperature.  If  the  pixel  is  determined  to  be  cloud  free,  its  brightness 
temperature  is  then  used  as  the  clear-scene  brightness  temperature  for  all  other  pixels  in 
the  frame.  To  determine  a  cloud-filled  pixel,  the  observed  brightness  temperature  of  the 
said  pixel  is  compared  to  the  predicted  clear-scene  brightness  temperature.  The 
difference  in  magnitude  of  brightness  temperature  determines  if  the  pixel  is  cloud-filled 
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or  cloudfree.  A  similar  method  is  used  for  the  visible  channel  when  available.  Threshold 
values  are  used  to  determine  the  cutoff  between  cloudy  and  clear  pixel  (HQ 
AFWA/DNMX  2011). 

The  algorithm  for  NOAA’s  AVHRR  exploits  the  multispectral  properties  of  the 
sensor  to  identify  cloud  or  clear  scene.  The  strength  of  the  AVHRR  is  its  six  channels 
designs,  which  are  listed  in  Table  9  with  their  respective  wavelengths  and  typical  use. 
Each  channel  differs  in  their  sensitivity  to  reflectivity  and  emissivity  properties  of  clouds 
and  clear  terrestrial  surfaces.  In  addition  to  the  sensors  data,  the  algorithm  uses  dear- 
scene  characterizations  of  the  terrestrial  background.  The  algorithm  utilizes  various 
techniques  to  include  straight  threshold  type  algorithms,  inter-channel  comparisons  and 
spectral  comparisons  between  the  terrestrial  surface  and  satellite  data.  A  suite  of  twelve 
tests  are  used  to  characterize  the  different  spectral  characteristics  of  clouds  and 
background  surfaces  to  determine  cloud-filled  or  clear  scene.  Each  test  is  based  on  one 
or  more  specific  spectral  signatures  that  compare  the  radiance  measurement  of  one  or 
more  channels,  and  fall  into  either  cloud  tests  or  background  tests  categories  (HQ 
AFWA/DNXM  2011). 
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Table  9. 


The  channels  below  are  from  the  AVHRR/3  sensor.  The  AVHRR  is  a 
radiation-detection  imager  that  can  be  used  for  remotely  determining  cloud 
cover  and  the  surface  temperature.  Note  that  the  term  surface  can  mean  the 
surface  of  the  Earth,  the  upper  surfaces  of  clouds,  or  the  surface  of  a  body  of 
water.  This  scanning  radiometer  uses  6  detectors  that  collect  different  bands 
of  radiation  wavelengths  as  shown  below.  Measuring  the  same  view,  this 
array  of  diverse  wavelengths,  after  processing,  permits  multi  spectral  analysis 
for  more  precisely  defining  hydrologic,  oceanographic,  and  meteorological 
parameters.  Comparison  of  data  from  two  channels  is  often  used  to  observe 
features  or  measure  various  environmental  parameters.  The  three  channels 
operating  entirely  within  the  infrared  band  are  used  to  detect  the  heat 
radiation  from  and  hence,  the  temperature  of  land,  water,  sea  surfaces,  and 
the  clouds  above  them.  Table  based  from 
NO  A  AS  IS  (2011). 


Channel 

number 

Resolution  at  nadir 
(km) 

Wavelength  (pm) 

Typical  use 

1 

1.09 

0.58-0.68 

Daytime  cloud  and  surface  mapping 

2 

1.09 

0.725-1.00 

Land-water  boundaries 

3a 

1.09 

1.58-1.64 

Snow  and  ice  detection 

3b 

1.09 

3.55-3.93 

Night  cloud  mapping,  sea  surface  temperature 

4 

1.09 

10.30-11.30 

Night  cloud  mapping,  sea  surface  temperature 

5 

1.09 

11.50-12.50 

Sea  surface  temperature 

There  are  nine  cloud  tests  and  three  background  tests,  which  are  summarized  in 
Table  10.  Different  tests  are  used  to  identify  clouds  under  different  conditions.  The  low 
clouds  and  fog  test  for  solar-illuminated  data  is  used  to  identify  water  droplets  based  low- 
level  cloud  when  the  scene  is  illuminated  by  sunlight,  and  the  non-illuminated  test  is  used 
during  nighttime.  Since  no  one  test  will  identify  all  the  clouds  in  a  scene,  the  cloud  tests 
must  be  used  in  combination  to  accurately  identify  all  cloud-filled  pixels.  The 
background  tests  are  unique  to  the  AVHRR  algorithm,  which  exploits  the  multispectral 
characteristics  of  AVHRR  data,  to  identify  snow  and  ice,  desert  and  sun  glint 
backgrounds.  These  tests  are  essential  because  clouds  and  surface  features  often  exhibit 
similar  spectral  signatures  in  the  visible  spectrum,  however;  a  positive  result  from  these 
tests  does  not  automatic  mean  a  cloudfree  pixel.  These  tests  identify  suspected  visible 
data;  the  infrared  cloud  tests  must  still  be  applied  to  determine  a  cloud-filled  pixel  (HQ 
AFWA/DNXM  2011). 
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Table  10. 


Cloud  analysis  test  for  the  NOAA  AVHRR  level  2  algorithm.  Table  from 

HQ  AFWA/DNXM  (2011). 


Visible 

IR  Data 

Test  Type 

Test  Name 

Major  Identifying  Class 

Data 

Low  cloud  and  fog  test  for  solar-illumination 
data 

Solar-illumination  liquid  water  cloud 

X 

Precipitation  cloud  test 

Cumulonimbus 

X 

X 

Cloud 

Thin  cirrus  cloud  test  for  solar-illumination 
data 

Solar-illumination  ice  cloud 

X 

X 

Visible  brightness  ratio 

Solar-illuminated  liquid  water  cloud 

X 

Single  channel  visible  brightness  test 

Solar-illuminated  liquid  water  cloud 

X 

Cold  cloud  test 

Mid-  to  high-level  optically  thick  water 
and  ice  cloud 

X 

Cirrus  cloud  test 

High-level  ice  cloud 

X 

Fog,  low  stratus  test  for  non-solar-illuminated 
data 

Non-solar-illuminated  liquid  water  cloud 

X 

Thin  cirrus  cloud  for  non-solar-illuminated 
data 

Non-solar-illuminated  ice  cloud 

X 

Sun  glint  background  test 

Water  surfaces  exhibiting  specular 

X 

X 

Background 

reflection 

Desert  background  test 

Highly  reflective  non-vegetated  land 
surfaces 

X 

X 

Snow/ice  cover  background  test 

Highly  reflective  snow  or  ice  covered 
land  and  water  surfaces 

X 

X 

There  are  three  algorithms  used  to  detection  clouds  from  geostationary  satellite 
data.  Geostationary  satellites  have  a  high  temporal  resolution  but  the  spatial  resolution  is 
degraded  due  to  its  altitude  (-36,000  km).  The  first  algorithm  takes  advantage  of  the 
high  temporal  resolution  to  identify  cloud-filled  pixels  by  testing  for  rapid  changes  in 
brightness  temperature  and  reflectance  values  in  pixels  representing  the  same 
geolocation.  The  pixels  that  exhibit  changes  in  radiance  values  greater  than  the  amount 
expected  for  clear  scene  from  frame  to  frame  are  identified  as  cloud-filled.  The  second 
algorithm  is  a  dynamic  threshold  algorithm  that  identifies  cloud  with  similar 
characteristics.  Cloud-filled  pixels  identified  through  the  temporal  difference  algorithm 
are  processed  by  the  dynamic  threshold  algorithm.  The  dynamic  threshold  algorithm 
identifies  maximum  and  minimum  brightness  temperatures  or  reflectance  within  a  grid 
cell,  which  are  used  to  define  threshold  values  for  cloud-filled  and  cloud-free  pixels 
remaining  within  the  grid  cell.  The  third  algorithm  uses  a  series  of  spectral 
discrimination  tests  similar  to  the  OLS  and  AVHRR  spectral  tests.  Not  all  geostationary 
satellite  data  are  the  same,  so  a  different  set  of  tests  may  need  to  be  ran  for  each  satellite 
system.  For  instance,  METOSAT  platforms  have  different  spectral  channels  than  GOES 
(HQ  AFWA/DNXM  2011).  Tables  11  and  12  summarize  the  spectral  channels  for 
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METEOSAT  and  GOES,  respectively.  The  resultant  dataset,  generated  for  each  satellite 
data  source,  is  called  Cloud  Data  Records  (CDR). 


Table  11.  Spectral  channels  and  bandwidth  for  METEOSAT  satellites.  Table  from 

EUMETSAT  (2011). 


Spatial  Resolution 

Channel _ Wavelength  (pm) _ at  nadir  (km) _ Remarks 

VIS  0.6  0.56-0.71  3 

VIS  0.8  0.74-0.88  3 

IR  1.6  1.50-1.78  3 

IR  3.9  3.48-4.36  3 


IR  8.7 

8.30-9.10 

3 

IR  10.8 

9.80-11.80 

3 

IR  12.0 

11.00-13.00 

3 

WV  6.2 

5.35-7.15 

3 

WV  7.3 

6.85-7.85 

3 

IR  9.7 

9.38-9.94 

3 

IR  13.4 

12.40-14.40 

3 

HRV 

0.5-0.9 

1 

Similar  to  AVHRR 
Similar  to  AVHRR 
Similar  to  AVHRR 
Similar  to  AVHRR 
New 

Similar  to  AVHRR 

Similar  to  AVHRR 

Water  vapor  channel 

Water  vapor  channel 

Ozone  absorption  channel  as  on  HIRS 

C02  absorption  channel  as  on  GOES-VAS  sounder 

High  Resolution  Visible  (HRV );  Broadband  visible  channel 


Table  12.  GOES  imager  channels.  Table  from  GOES  Imager  Channel  Notation  (201 1). 


Channel 

number 

Resolution  at  nadir 
or  IFOV*  (km) 

Wavelength 

(pm) 

Remarks 

1 

1 

0.55-0.75 

Visible 

2 

4 

3.80-4.00 

Shortwave  Infrared 

3 

8 

6.50-7.00 

Moisture 

4 

4 

10.20-11.20 

Infrared  1 

5 

4 

11.50-12.50 

Infrared  2 

*  Instantaneous  Field  of  View 


C.  LEVEL  3 

Level  three  is  where  the  satellite  pixels  are  gridded  onto  AFWA’s  standard  Polar- 
stereographic  grid  at  “16th  mesh”  with  a  horizontal  resolution  of  24  km  (true  at  60° 
latitude).  Pixels  are  assembled  into  the  16th  mesh  grid  cells  by  computing  the  coordinates 
that  correspond  to  the  latitude  and  longitude  of  each  pixel  (Hoke  et  al.,  1981  Rev.  March 
1985).  A  detailed  description  of  the  Polar-stereographic  grid  is  provided  in  Map 
Projections  and  Grid  System  for  Meteorological  Applications,  AFGWC  Technical  Notes 
79/003  (Hoke  et  al.,  1981  Rev.  March  1985).  The  cloud  layers  are  identified  through  the 
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Long  Wave  Infrared  (LWIR)  brightness  temperature  data  contained  within  each  grid  cell. 
A  clustering  algorithm  clusters  pixels  of  similar  brightness  characteristics  to  identify 
potential  layer  separations.  Statistical  procedures  are  applied  to  the  grid  cell  to  limit  the 
identified  layers  to  four.  Once  the  layers  are  identified,  cloud  top  temperatures  are 
compared  against  vertical  temperature  information  from  the  Nation  Centers  for 
Environmental  Prediction  (NCEP)  Global  Forecast  System  (GFS)  model  to  assign  a 
cloud  top  height.  The  cloud  top  height  and  temperature  information  is  used  with  the 
visible/LWIR-count  variance,  from  the  background  surface  temperature  model  employed 
in  level  two,  to  assign  each  layer  to  one  of  nine  different  cloud  types  listed  in  Table  13. 
Along  with  each  derived  cloud  type  is  a  climatological  cloud  thickness  that  is  subtracted 
from  the  cloud  top  height  to  determine  the  cloud  base  height  (HQ  AFWA/DNXM  2011). 
At  the  end  of  level  three,  each  satellite  family  (e.g.,  DMSP,  NOAA,  Geostationary)  have 
a  common  gridded  cloud  mask  that  consists  of  cloud  fraction  up  to  four  layers.  The 
cloud  masking  also  includes  cloud  type,  and  cloud  top/base  heights.  These  datasets  are 
called  Gridded  Data  Records  (GDR). 


Table  13.  WWMCA  default  cloud  thickness  according  to  height.  Cloud  thickness  is 
based  on  climatology.  Table  from  HQ  AFWA/DNXM  (201 1). 


Type  Code 

Cloud  Type 

Thickness  (m) 

1 

Cumulonimbus  (Cb) 

6300 

2 

Strata  (St) 

600 

3 

Stratocumulus  (Sc) 

1800 

4 

Cumulus  (Cu) 

1200 

5 

Altostratus  (As) 

1200 

6 

Nimbostratus  (Ns) 

2100 

7 

Altocumulus  (Ac) 

1800 

8 

Cirrostratus  (Cs) 

1800 

9 

Cirrus  (Ci) 

900 

Fevel  three  processing  also  includes  hourly  global  surface  and  upper  air  based 
data,  METARS  or  SYNTOPIC  type  formats,  which  contain  fractional  cloud  coverage  and 
cloud  base  heights  from  the  World  Meteorological  Organization.  These  conventional 
observations  are  combined  with  the  satellite  data  to  determine  the  cloud  mask,  cloud 
type,  and  cloud  top/base  heights. 
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D.  LEVEL  4 


Level  four  is  where  the  satellite  family  GDRs  and  conventional  surface 
observations  are  merged  into  a  single  global  analysis  of  cloud  cover  information.  One 
problem  that  arises  in  level  four  is  that  the  independent  gridded  analyses  have  different 
valid  times  because  the  satellites  input  their  data  into  CDFS  II  at  different  times.  Each 
independent  gridded  analysis  has  strengths  and  weaknesses.  For  example,  the  polar- 
orbiters  (DMSP  and  NOAA  satellites)  derived  analyses  have  greater  accuracy  from  the 
spatial  resolution  (polar  satellites  are  in  a  lower  orbit,  -800  km);  however,  the  temporal 
resolution  is  course,  usually  passing  over  a  particular  region  one  or  two  times  a  day. 
Geostationary  satellites  analyses  have  a  finer  temporal  resolution,  every  30  minutes,  but 
spatial  resolution,  or  instantaneous  field  of  view  (IFOV),  varies  from  1  to  8  km 
depending  on  the  channel.  See  the  resolution  at  nadir  column  in  Table  12  for  each 
channel’s  IFOV.  The  timeliness  and  accuracy  of  the  observations  is  a  major  concern 
when  merging  the  data  into  a  one  global  analysis  (HQ  AFWA/DNXM  2011). 

Integration  of  total  cloud  amounts  supersedes  integration  of  layered  quantities 
since  total  cloud  fraction  estimates  are  more  accurate  than  individual  layer  fraction  due  to 
the  sample  size  of  total  cloud  amount  is  far  greater  than  the  layered  cloud  amounts  (HQ 
AFWA/DNXM  2011).  Bartlett  (2009)  explains  how  the  total  cloud  amount  analysis 
works: 


Total  cloud  fraction  is  then  set  to  either  100  or  0  percent,  respectively.  If 
neither  analysis  is  completely  cloud-filled  or  completely  cloud-free,  then 
the  error  for  each  analysis  is  estimated.  The  estimated  errors  for  the 
analyses  are  compared  to  one  another  to  see  if  the  most  recent  analysis 
also  has  the  lowest  estimated  error.  Optimum  interpolation  (OI)  occurs 
when  one  analysis  cannot  be  chosen  as  the  most  accurate.  OI  maintains  a 
blended  estimate  of  total  cloud  fraction  from  multiple  input  analyses. 
Weighting  functions  for  the  OI  are  based  on  the  estimated  analysis  errors 
which  are  computed  for  each  individual  analysis.  Analysis  errors  are 
defined  as  an  initial  analysis  error  plus  an  additional  error  growth  function 
which  grows  linearly  with  time.  The  error  growth  function  is  a  tunable 
parameter  that  analysts  can  adjust  to  correct  for  inconsistencies. 

When  total  cloud  amount  is  completed  the  other  cloud  parameters  are  merged. 

Rules  applied,  to  determine  which  analysis  is  superior,  in  the  layered  analysis  are  similar 
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to  the  total  cloud  amount;  however,  if  multiple  timely  analyses  have  100%  cloud  cover  or 
it  is  determined  that  an  01  technique  is  necessary.  The  integration  of  layered  cloud 
amounts  undergoes  a  more  extensive  algorithm.  Most  likely  the  individual  analyses  will 
have  varying  vertical  distributions  of  cloud  and  cloud  type  due  to  the  differences  in 
sensor  characteristics  for  each  satellite  family.  The  more  complex  algorithm  determines 
which  analyses  is  the  most  accurate  and  designates  that  analyses  as  the  master  template 
for  which  all  other  timely  analyses  are  merged  on.  This  process  impacts  discrete  values 
such  as  the  number  of  cloud  layers  and  cloud  types  because  these  when  integrated  they 
will  assume  the  values  of  the  master  template.  The  01  procedure  is  used  for  varying 
layered  cloud  faction  and  cloud  top  temperature.  The  01  process  combines  layers  that 
closely  match  in  cloud  top  temperatures  and  determines  the  layered  cloud  fraction. 
Special  cloud  algorithms  have  been  designed  for  certain  satellite  sensors  to  enhance 
detection  of  low  level  stratus  and  cirrus.  These  special-case  clouds  are  verified  against 
the  integrated  analysis  to  be  certain  that  the  analysis  is  accurate,  and  are  effective  in 
showing  the  persistence  of  the  observations  in  the  subsequent  integration  analysis  (HQ 
AFWA/DNXM  2011). 

All  the  output  variables  are  placed  in  a  GriB  file  (a  collection  of  individual  self- 
containing  records,  and  the  individual  records  themselves  can  stand  alone  as  meaningful 
data)  and  is  published  as  the  Worldwide  Merge  Cloud  Analysis  (WWMCA). 


75 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


76 


APPENDIX  B.  SPREADSHEET  STRUCTURE  AND  FORMULAS  USED  TO  VERIFY  WWMCA  WITH 

CLOUDSAT 


Sheet 

Section 

Column 

Name 

Formula 

Remarks 

A 

WWMCA  valid  time 

Imported  from  CSV  file 

B 

Cloudsat  time 

Imported  from  CSV  file 

C 

Cloudsat  Lat 

Imported  from  CSV  file 

D 

Cloudsat  Lon 

Imported  from  CSV  file 

E 

WWMCA  Lat 

Imported  from  CSV  file 

F 

WWMCA  Lon 

Imported  from  CSV  file 

G 

Distance  Between  WWMCA  and  Cloudsat  Points  (m) 

Imported  from  CSV  file 

H 

Cloudsat  Max  Mask 

Imported  from  CSV  file 

I 

Cloudsat  Mask  Text  (20  threshold) 

Imported  from  CSV  file 

Q 

J 

Cloudsat  Occurrence  Value  (20  threshold) 

Imported  from  CSV  file 

T5 

K 

Cloudsat  Mask  Text  (30  threshold) 

Imported  from  CSV  file 

!/3 

QJ 

U 

L 

Cloudsat  Occurrence  Value  (30  threshold) 

Imported  from  CSV  file 

O 

Ph 

M 

WWMCA  Valid  Time 

Imported  from  CSV  file 

N 

WWMCA  i 

Imported  from  CSV  file 

O 

WWMCA  j 

Imported  from  CSV  file 

P 

Hemisphere 

Imported  from  CSV  file 

Q 

WWMCA  Layer  1  Cloud  Cover  (%) 

Imported  from  CSV  file 

R 

WWMCA  Layer  1  Cloud  Type 

Imported  from  CSV  file 

S 

WWMCA  Layer  1  Height  of  Cloud  Base  (m) 

Imported  from  CSV  file 

T 

WWMCA  Layer  1  Height  of  Cloud  Top  (m) 

Imported  from  CSV  file 

U 

WWMCA  Layer  2  Cloud  Cover  (%) 

Imported  from  CSV  file 

V 

WWMCA  Layer  2  Cloud  Type 

Imported  from  CSV  file 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 

W 

WWMCA  Layer  2  Height  of  Cloud  Base  (m) 

Imported  from  CSV  file 

X 

WWMCA  Layer  2  Height  of  Cloud  Top  (m) 

Imported  from  CSV  file 

Y 

WWMCA  Layer  3  Cloud  Cover  (%) 

Imported  from  CSV  file 

Z 

WWMCA  Layer  3  Cloud  Type 

Imported  from  CSV  file 

AA 

WWMCA  Layer  3  Height  of  Cloud  Base  (m) 

Imported  from  CSV  file 

AB 

WWMCA  Layer  3  Height  of  Cloud  Top  (m) 

Imported  from  CSV  file 

C3 

C3 

AC 

WWMCA  Layer  4  Cloud  Cover  (%) 

Imported  from  CSV  file 

— 

T5 

AD 

WWMCA  Layer  4  Cloud  Type 

Imported  from  CSV  file 

0> 

cc 

5*5 

AE 

WWMCA  Layer  4  Height  of  Cloud  Base  (m) 

Imported  from  CSV  file 

u 

o 

AF 

WWMCA  Layer  4  Height  of  Cloud  Top  (m) 

Imported  from  CSV  file 

c- 

AG 

WWMCA  Total  Cloud  Cover  (%) 

Imported  from  CSV  file 

AH 

WWMCA  Cloud  Cover  Bin 

Imported  from  CSV  file 

AI 

WWMCA  Pixel  Age  (min) 

Imported  from  CSV  file 

AJ 

Cloudsat  Total  Cloud  Cover  (%  -  20  threshold) 

Imported  from  CSV  file 

AK 

Cloudsat  Cloud  Eventing  (20  threshold) 

Imported  from  CSV  file 

AL 

Cloudsat  Total  Cloud  Cover  (%  -  30  threshold) 

Imported  from  CSV  file 

AM 

Cloudsat  Cloud  Eventing  (30  threshold) 

Imported  from  CSV  file 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 

A 

CSV  Line  Number 

Tracks  CSV  line  numbers  in  this  workbook 

B 

WWMCA  valid  time 

=’Processed  Data'!A3 

From  Processed  Data  sheet  Column  A 

C 

WWMCA  latitude 

='Processed  Data'!E3 

From  Processed  Data  sheet  Column  E 

D 

WWMCA  longitude 

='Processed  Data'!F3 

From  Processed  Data  sheet  Column  F 

CS 

a 

E 

WWMCA  i 

='Processed  Data'!N3 

From  Processed  Data  sheet  Column  N 

F 

WWMCA  j 

='Processed  Data'!03 

From  Processed  Data  sheet  Column  O 

Q 

'd 

G 

Cloudsat  time 

='Processed  Data'!B3 

From  Processed  Data  sheet  Column  B 

QJ 

<*> 

Vl 

0> 

CJ 

o 

H 

WWMCA  total  cloud  amount 

(%) 

='Processed  Data'!AG3 

From  Processed  Data  sheet  Column  AG 

s- 

CU 

I 

WWMCA  cloud  cover  event 

='Processed  Data'!AH3 

From  Processed  Data  sheet  Column  AH 

C3 

£ 

o 

u 

ta 

J 

Cloudsat-20  total  cloud 

amount  (%  ) 

='Processed  Data'!AJ3 

From  Processed  Data  sheet  Column  AJ 

"S 

— 

K 

Cloudsat-30  total  cloud 

amount  (%) 

='Processed  Data'!AL3 

From  Processed  Data  sheet  Column  AL 

s- 

L 

Cloudsat-20  cloud  cover  event 

='Processed  Data'!AK3 

From  Processed  Data  sheet  Column  AK 

E 

M 

Cloudsat-30  cloud  cover  event 

='Processed  Data'!AM3 

From  Processed  Data  sheet  Column  AM 

N 

Filtered  WWMCA  cloud  cover 
event  (Cloudsat-20) 

=IF(J3"n/a  (less  than  6  close  points)", 

B) 

Filtered  out  WWMCA  cloud  cover  bin  values  that 
corresponded  to  Cloudsat-20  cells  that  had  "n/a 
(less  than  6  close  points)"  and  replace  with  "" 

d 

d 

O 

T3 

O 

Filtered  WWMCA  cloud  cover 
event  (Cloudsat-30) 

=IF(K3"n/a  (less  than  6  close  points)", 
13) 

Filtered  out  WWMCA  cloud  cover  bin  values  that 
corresponded  to  Cloudsat-30  cells  that  had  "n/a 
(less  than  6  close  points)"  and  replace  with  "" 

0> 

a> 

P 

Filtered  Cloudsat-20  total 

=IF(J3"n/a  (less  than  6  close  points)",  "", 

Filtered  out  the  "n/a  (less  than  6  close  points)"  from 

cloud  amount  (%) 

J3) 

Cloudsat-20  total  cloud  amount  and  replace  with  "" 

n 

Filtered  Cloudsat-30  total 

=IF(K3"n/a  (less  than  6  close  points)", 

Filtered  out  the  "n/a  (less  than  6  close  points)"  from 

V 

cloud  amount  (%) 

”",K3) 

Cloudsat-30  total  cloud  amount  and  replace  with  "" 

R 

Filtered  Cloudsat-20  cloud 

cover  event 

=IF(L3=-1,  L3) 

Filtered  out  values  of  -1  from  Cloudsat-20  cloud 

cover  event 
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S 

Filtered  Cloudsat-30  cloud 

cover  event 

=IF(M3=-1,  M3) 

Filtered  out  values  of  -1  from  Cloudsat-30  cloud 

cover  event 

WWMCA  cloud  cover  event 

A 

A 

T 

(Cloudsat-20)  w/  values  of  '2' 

=IF(N3=2, N3) 

removed 

— 

T3 

WWMCA  cloud  cover  event 

S- 

o> 

U 

(Cloudsat-30)  w/  values  of  '2' 

=IF(03=2, 03) 

-4-1 

removed 

V 

Cloudsat-20  cloud  cover  event 
w/  values  of  '2'  removed 

=IF(R3=2,  R3) 

w 

Cloudsat-30  cloud  coverevent 
w/  values  of  '2'  removed 

=IF(S3=2, S3) 

eg 

S3 

T3 

%Diff  in  WWMCA  & 

D 

O 

X 

Cloudsat-20  total  cloud 

=(H3-P3)/H3 

Q 

G 

amount  (%) 

CJ 

u 

•*-* 

%Diff  in  WWMCA  & 

QJ 

o 

H 

Y 

Cloudsat-30  total  cloud 

=(H3-Q3)/H3 

E 

©  S3 

amount  (%) 

u  © 

g  s 

Filtered  %  Diff  in  WWMCA  & 

Filtered  out  values  of  ’#DIV/0!'  from  %  diff  in 

a>  H 

SN  «< 

z 

Cloudsat-20  total  cloud 

WWMCA  &  Cloudsat-20  total  cloud  amount  with  a 

83 

2 

amount  (%) 

blank  by  find/replace  function. 

S3 

Filtered  %  Diff  in  WWMCA  & 

Filtered  out  values  of  ’#DIV/0!'  from  %  diff  in 

u 

AA 

Cloudsat-30  total  cloud 

WWMCA  &  Cloudsat-30  total  cloud  amount  by 

— 

amount  (%) 

find/replace  function. 

A 

AZ 

Cloudsat  time  for  Flagging 

=G3 

time  in  decimal  form 

ie  of  D 
Flags 

BA 

Day/Night  Flag 

=IF(AND(AZ3>0.625,  AZ3<1),  "Day", 
"Night") 

see  Figure  12  for  other  daytime  ranges 

S 

H 

BB 

Polar  Night  Flag 

IFfProcessed  Data"!C3>=70,  "Night", 
BA3) 

Used  for  Cloudsat  latitudes  above  70°N.  For  Polar 
Day,  replace  "Night"  with  "Day." 
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3  ^ 

AB 

Event  3  (Cloudsat-20)  Cell  A 
(Hit) 

=IF(AND(N3=3,  R3=3),  "Hit",  "") 

AC 

Event  3  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF(AND(N3=3,  R3<2),  "FA",  "") 

su  O  'S 
§ 

’■§  oU 

AD 

Event  (Cloudsat-20)  Cell  C 
(Miss) 

=IF(AND(N3<2,  R3=3),  "Miss", "") 

AE 

Event  3  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF(AND(N3<2,  R3<2),  "Reject",  "") 

cs 

u 

<© 

AF 

Event  2  (Cloudsat-20)  Cell  A 
(Hit) 

=IF(AND(N3=2,  R3=2),  "Hit",  "") 

a 

— 

A 

Q 

AG 

Event  2  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF(AND(N3=2,  V3<3),  "FA", "") 

s- 

o> 

< 

Qm 

o 

AH 

Event  2  (Cloudsat-20)  Cell  C 
(Miss) 

=IF(AND(T3<3,  R3=2),  "Miss",  "") 

t/J  ® 

0)  <N 

s  ^ 

AI 

Event  2  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF(AND(T3<3,  V3<3),  "Reject",  "") 

C/5  ^3 

0 i  D 

3  3 

AJ 

Event  1  (Cloudsat-20)  Cell  A 
(Hit) 

=IF(AND(N3=1,  R3=l),  "Hit",  "") 

H 

o 

S3 

a> 

AK 

Event  1  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF(AND(N3=1,  R3>1),  "FA",  "") 

ox 

#g 

V-* 

S3 

AL 

Event  1  (Cloudsat-20)  Cell  C 
(Miss) 

=IF(AND(N3>1,  R3=l),  "Miss", "") 

u 

AM 

Event  1  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF(AND(N3>1,  R3>1),  "Reject",  "") 
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AN 

Event  3  (Cloudsat-30)  Cell  A 
(Hit) 

=IF(AND(P33,  S33),  "Hit", "") 

o 

rn 

AO 

Event  3  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF(AND(P33,  S3<2),  "FA",  "") 

■m 

a 

v 

© 

o 

AP 

Event  (Cloudsat-30)  Cell  C 
(Miss) 

=IF(AND(P3<2,  S33),  "Miss",  "") 

U 

s- 

£ 

AQ 

Event  3  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF(AND(P3<2,  S3<2),  "Reject", "") 

C3 

-m 

© 

P 

— 

< 

AR 

Event  2  (Cloudsat-30)  Cell  A 
(Hit) 

=IF(AND(P32,  S32),  "Hit", "") 

TS 

© 

U 

© 

Cm 

O 

V 

© 

Sh 

AS 

Event  2  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF(AND(P32,  W3<3),  "FA", "") 

a 

© 

© 

t/5 

_© 

AT 

Event  2  (Cloudsat-30)  Cell  C 
(Miss) 

=IF(AND(U3<3,  S32),  "Miss",  "") 

»© 

© 

H 

AU 

Event  2  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF(AND(U3<3,  W3<3),  "Reject",  "") 

a 

© 

ox 

AV 

for  bin  (1  -30)  Cell  A  (Hit) 

=IF(AND(P31,  S31),  "Hit", "") 

*-M 

© 

© 

U 

AW 

for  bin  (1  -30)  Cell  B  (False 
Alarm) 

=IF(AND(P31,  S3>1),  "FA",  "") 

AX 

for  bin(l  -30)  Cell  C  (Miss) 

=IF(AND(P3>1,  S31),  "Miss",  "") 

AY 

for  bin  (1  -  30)  Cell  D  (Correct 
Rejection) 

=IF(AND(P3>1,  S3>1),  "Reject", "") 
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BC 

Event  3  (Cloudsat-20)  Cell  A 
(Hit) 

=IF($BB3”Day",  AB3,  "") 

o 

BD 

Event  3  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3”Day",  AC3,  "") 

1 

a 

n 

h3 

3 

BE 

Event  (Cloudsat-20)  Cell  C 
(Miss) 

=IF($BB3"Day",  AD3,  "") 

o 

0 

£ 

BF 

Event  3  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Day",  AE3,  "") 

cs 

a 

Q 

s 

c« 

Q 

«w 

O 

BG 

BH 

Event  2  (Cloudsat-20)  Cell  A 
(Hit) 

Event  2  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3"Day",  AEG,  "") 

=IF($BB3”Day",  AG3,  "") 

u 

QJ 

E 

t/3 

© 

S- 

o 

o 

Ifl 

3 

cs 

H 

u 

fl 

© 

OX) 

#c 

a 

© 

BI 

BJ 

Event  2  (Cloudsat-20)  Cell  C 
(Miss) 

Event  2  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3”Day",  AH3,  "") 

=IF($BB3”Day",  AI3,  "") 

BK 

BL 

Event  1  (Cloudsat-20)  Cell  A 
(Hit) 

Event  1  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3”Day",  AJ3,  "") 

=IF($BB3”Day",  AK3,  "") 

u 

BM 

Event  (Cloudsat-20)  Cell  C 
(Miss) 

=IF($BB3”Day",  AL3,  "”) 

BN 

Event  1  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Day",  AM3,  "") 
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BO 

Event  3  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3”Day",  AN3,  '"') 

o 

rr> 

BP 

Event  3  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3”Day",  A03,  "") 

+* 

a 

on 

a 

BQ 

Event  (Cloudsat-30)  Cell  C 
(Miss) 

=IF($BB3”Day",  AP3,  "") 

o 

0 

u 

© 

BR 

Event  3  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Day",  AQ3,  "") 

Cm 

QJ 

S 

*-m 

BS 

Event  2  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3"Day",  AR3,  "") 

a 

Q 

T3 

a 

Q 

®M 

© 

BT 

Event  2  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3"Day",  AS3,  "") 

0> 

U 

QJ 

t « 

© 

S- 

© 

o 

BU 

Event  2  (Cloudsat-30)  Cell  C 
(Miss) 

=IF($BB3”Day",  AT3, "") 

C/2 

3 

CS 

BV 

Event  2  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Day",  AU3,  ”") 

H 

© 

= 

BW 

Event  1  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3”Day",  AV3,  "") 

DU 

C 

*-m 

a 

BX 

Event  1  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3"Day",  AW3,  "") 

u 

BY 

Event  1  (Cloudsat-30)  Cell  C 
(Miss) 

=IF($BB3"Day",  AX3,  "") 

BZ 

Event  1  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Day",  AY3,  "") 
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CA 

Event  3  (Cloudsat-20)  Cell  A 
(Hit) 

=IF($BB3"Night",  AB3,  "") 

o 

r* 

1 

CB 

Event  3  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3"Night",  AC3,  "") 

A 

C/2 

d 

s 

o 

CC 

Event  (Cloudsat-20)  Cell  C 
(Miss) 

=IF($BB3"Night",  AD3,  "") 

u 

■- 

a 

CD 

Event  3  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night",  AE3,  "") 

QJ 

S 

*-M 

■M 

CE 

Event  2  (Cloudsat-20)  Cell  A 
(Hit) 

=IF($BB3"Night",  AF3,  "") 

■M 

ft 

— 

d 

gf 

55 

CM 

CF 

Event  2  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3"Night”,  AG3,  "") 

U 

QJ 

•M 

a 

fa 

o 

1/2 

a> 

s- 

o 

QJ 

W 

QJ 

3 

ft 

H 

CJ 

S3 

CG 

CH 

Event  2  (Cloudsat-20)  Cell  C 
(Miss) 

Event  2  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night",  AH3, "") 

=IF($BB3"Night",  AI3,  "") 

Cl 

Event  1  (Cloudsat-20)  Cell  A 
(Hit) 

=IF($BB3"Night",  AJ3,  "") 

a> 

OX) 

#g 

fa 

fl 

CJ 

Event  1  (Cloudsat-20)  Cell  B 
(False  Alarm) 

=IF($BB3"Night”,  AK3,  "") 

o 

U 

CK 

Event  1  (Cloudsat-20)  Cell  C 
(Miss) 

=IF($BB3"Night”,  AL3,  "") 

CL 

Event  1  (Cloudsat-20)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night”,  AM3,  "") 
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CM 

Event  3  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3"Night",  AN3,  "") 

o 

i 

CN 

Event  3  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3"Night",  A03,  "") 

S3 

Gfi 

s 

o 

CO 

Event  (Cloudsat-30)  Cell  C 
(Miss) 

=IF($BB3"Night",  AP3,  "") 

0 

L. 

<2 

CP 

Event  3  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night",  AQ3,  "") 

0> 

s 

*-M 

■M 

CQ 

Event  2  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3"Night",  AR3,  "") 

■M 

ft 

— 

T3 

gf 

2 

CM 

CR 

Event  2  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3"Night”,  AS3,  "") 

U 

QJ 

•M 

a 

fa 

o 

1/3 

a> 

s- 

o 

u 

w 

QJ 

3 

ft 

H 

CJ 

S3 

CS 

CT 

Event  2  (Cloudsat-30)  Cell  C 
(Miss) 

Event  2  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night",  AT3,  ”") 

=IF($BB3"Night",  AU3,  "") 

CU 

Event  1  (Cloudsat-30)  Cell  A 
(Hit) 

=IF($BB3"Night",  AV3,  "") 

a> 

OX) 

#g 

*M 

fi 

CV 

Event  1  (Cloudsat-30)  Cell  B 
(False  Alarm) 

=IF($BB3"Night”,  AW3,  "") 

o 

U 

CW 

Event  1  (Cloudsat-30)  Cell  C 
(Miss) 

=IF($BB3"Night”,  AX3,  "") 

CX 

Event  1  (Cloudsat-30)  Cell  D 
(Correct  Rejection) 

=IF($BB3"Night”,  AY3,  "") 
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APPENDIX  C.  RESULTS  FOR  WWMCA  BOXES  22  AND  29  FOR 
CLOUDSAT  CLOUD  MASK  VALUE  OF  20  USED  FOR  CLOUD/NO¬ 
CLOUD  THRESHOLD 


A.  WWMCA  BOX  22 


January  Daytime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


April  Daytime  Occurrences  WWMCA  (22)-  Cloudsat  (20)  2010 


A  B  C  D 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


60000 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


January  Nighttime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


April  Nighttime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


A  B  C  D 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


60000 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


January  All  Day  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


A  B  C  D 


April  All  Day  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


120000 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


Figure  31.  Number  of  occurrences  in  January  and  April  2010  for  WWMCA  Box  22  for 
all  three  cloud  cover  events  (cloud,  probable  cloud,  and  no-cloud)  using  a 
Cloudsat  cloud  mask  value  of  20  for  cloud/no-cloud  threshold.  The  “cloud” 
event  (blue)  represents  the  “definite  cloud”  event.  The  horizontal  axes  are  the 
occurrence  categories:  A — Hits;  B — False  Alarms;  C — Misses;  and  D — 
Correct  Rejections.  The  top  panels  are  the  occurrences  for  daytime,  the 
middle  panels  are  nighttime  occurrences,  and  the  bottom  panels  are  the 
combined  results  of  both  daytime  and  nighttime  occurrences. 
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July  Daytime  Occurences  WWMCA  (22)  -  Cloudsat  (20)  2010 


■  ProbableCloud 


October  Daytime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 
60000 


■  Pro  table  Cloud 


July  Nightime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 


■  Cloud 

■  Probable  Cloud 

■  No  Cloud 


October  Nighttime  Occurrences  WWMCA  (22)  -  Cloudsat  (20)  2010 
60000 


■  ProbableCloud 


October  All  Day  Occurrences  WWMCA  (22)  -  Cloudsay  (20)  2010 


■  Cloud 

■  ProbableCloud 

■  No  Cloud 


Figure  32.  Number  of  2  x  2  contingency  table  occurrences  in  July  and  October  2010  for 
WWMCA  Box  22  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Daytime  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(22)  -  Cloudsat  (20)  2010 


Jan  Apr  Jul  Oct 


■  WWMCACIoud 

■  Cloud  sat_Ck>ud 

■  WWMCA_Prob  CkJ 

■  Ooudsat_Prob  CkJ 

■  WWMCA_No  Cloud 

■  Cloud  sat  _No  Cloud 


Nighttime  Marginal  Distributions  of  Cloud  Cover  Events  for 
WWMCA  (22)  -  Cloudsat  (20)  2010 


■  WWMCACIoud 

■  C  lo  ud  sat  _C  loud 

■  WWMCA_Prob  CkJ 

■  Ck>udsat_Prob  Cld 

■  WWMCA_No  Cloud 

■  Cloud  sat_No  Cloud 


All  Day  Marginal  Distributions  of  Cloud  Cover  Events  for  WWMCA 
(22)  -  Cloudsat  (20)  2010 


Jan  Apr  Jul  Oct 


■  WWMCA_Ck>ud 

■  Cloud  sat_Ck>ud 

■  WWMCA_Prob  CkJ 

■  Cloud  sat_Prob  Ckd 

■  WWMCA_No  Cloud 

■  Ooudsat_No  Cloud 


Figure  33.  Marginal  distributions  of  cloud  cover  events  for  WWMCA  and  Cloudsat  for 
WWMCA  Box  22:  Definite-Cloud,  Probable-Cloud,  and  No-Cloud).  The 
calculation  of  the  marginal  distributions  is  shown  in  Table  7.  The  top  panel 
shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data,  and 
the  bottom  panel  for  both  daytime  and  nighttime  data.  Results  based  on  using 
a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Daytime  Hit  Rate  WWMCA  (22)  -  Cloudsat  (20)  2010 


Jan  Apr  Jul  Oct 


>  Cloud 

—•“Probable  Cloud 
•  No  Cloud 


Nighttime  Hit  Rate  WWMCA  (22)  -  Cloudsat  (20)  2010 


Jan  Apr  Jul  Oct 


•  Definite  Cloud 
^^—Probable  Cloud 
—•—No  Cloud 


All  Day  Hit  Rate  WWMCA  (22)  -  Cloudsat  (20)  2010 


-Cloud 

— ProbableCloud 
-No  Cloud 


Apr 


Figure  34.  WWMCA  probability  of  detection  (POD),  or  hit  rate  (H),  for  WWMCA  Box 
22  for  all  three  cloud  cover  events:  Definite-Cloud  (blue),  Probable-Cloud 
(red),  and  No-Cloud  (green).  The  top  panel  shows  the  results  for  daytime 
data,  the  middle  panel  for  nighttime  data,  and  the  bottom  panel  for  both 
daytime  and  nighttime  data.  Results  based  on  using  a  Cloudsat  cloud  mask 
value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  35.  WWMCA  false  alarm  ratios  for  WWMCA  Box  22  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  36.  WWMCA  proportion  correct  (PC)  for  WWMCA  Box  22  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  37.  WWMCA  threat  scores  (TS)  for  WWMCA  Box  22  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  38.  WWMCA  bias  (B)  for  WWMCA  Box  22  for  all  three  cloud  cover  events: 

Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green).  The  top 
panel  shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data, 
and  the  bottom  panel  for  both  daytime  and  nighttime  data.  The  heavy  dark 
line  at  B  =  1  indicates  an  unbiased  analysis.  Values  greater  (less)  than  1 
indicate  a  cloud  cover  event  that  was  over  (under)  analyzed.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 


94 


Daytime  HSS  WWMCA  (22)  -  Cloudsat  (20)  2010 


£ 

8 


<S) 


•  Cloud 

^^^Probable  Cloud 

•  No  Cloud 


Nighttime  HSS  WWMCA  (22)  -  Cloudsat  (20)  2010 


»  Cloud 

ro  ba  bl  e  Cloud 
■  No  Ctoud 


All  Day  HSS  WWMCA  (22)  >  Cloudsat  (20)  2010 


»  Ctoud 

^»“P  ro  ba  bl  e  Cloud 
■  No  Cloud 


Figure  39.  WWMCA  Heidke  skill  score  (HSS)  for  WWMCA  Box  22  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 

The  heavy  dark  line  at  the  zero  value  indicates  that  the  WWMCA  is 
equivalent  to  reference  analyses  (random  analyses  statistically  independent  of 
observations).  A  perfect  WWMCA  cloud  analysis  would  have  a  value  of  one. 
Negative  values  indicate  WWMCA  performs  worse  than  reference  analyses. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  40.  Number  of  2  x  2  contingency  table  occurrences  in  January  and  April  2010  for 
WWMCA  Box  29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  41.  Number  of  2  x  2  contingency  table  occurrences  in  July  and  October  2010  for 
WWMCA  Box  29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue), 
Probable-Cloud  (red),  and  No-Cloud  (green).  On  the  horizontal  axes:  A 
indicates  Hits,  B  indicates  False  Alarms,  C  indicates  Misses,  and  D  indicates 
Correct  Rejections.  The  top  panels  shows  the  daytime  occurrences,  the 
middle  panels  shows  the  nighttime  occurrences,  and  the  bottom  panels  show 
the  results  for  both  daytime  and  nighttime  occurrences.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  42.  Marginal  distributions  of  cloud  cover  events  for  WWMCA  and  Cloudsat  for 
WWMCA  Box  29:  Definite-Cloud,  Probable-Cloud,  and  No-Cloud).  The 
calculation  of  the  marginal  distributions  is  shown  in  Table  7.  The  top  panel 
shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data,  and 
the  bottom  panel  for  both  daytime  and  nighttime  data.  Results  based  on  using 
a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  43.  WWMCA  probability  of  detection  (POD),  or  hit  rate  (H),  for  WWMCA  Box 
29  for  all  three  cloud  cover  events:  Definite-Cloud  (blue),  Probable-Cloud 
(red),  and  No-Cloud  (green).  The  top  panel  shows  the  results  for  daytime 
data,  the  middle  panel  for  nighttime  data,  and  the  bottom  panel  for  both 
daytime  and  nighttime  data.  Results  based  on  using  a  Cloudsat  cloud  mask 
value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  44.  WWMCA  false  alarm  ratios  for  WWMCA  Box  29  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  45.  WWMCA  proportion  correct  (PC)  for  WWMCA  Box  29  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  46.  WWMCA  threat  scores  (TS)  for  WWMCA  Box  29  for  all  three  cloud  cover 
events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green). 

The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel  for 
nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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Figure  47.  WWMCA  bias  (B)  for  WWMCA  Box  29  for  all  three  cloud  cover  events: 

Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud  (green).  The  top 
panel  shows  the  results  for  daytime  data,  the  middle  panel  for  nighttime  data, 
and  the  bottom  panel  for  both  daytime  and  nighttime  data.  The  heavy  dark 
line  at  B  =  1  indicates  an  unbiased  analysis.  Values  greater  (less)  than  1 
indicate  a  cloud  cover  event  that  was  over  (under)  analyzed.  Results  based  on 
using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no-cloud  threshold. 
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Figure  48.  WWMCA  Heidke  skill  score  (HSS)  for  WWMCA  Box  29  for  all  three  cloud 
cover  events:  Definite-Cloud  (blue),  Probable-Cloud  (red),  and  No-Cloud 
(green).  The  top  panel  shows  the  results  for  daytime  data,  the  middle  panel 
for  nighttime  data,  and  the  bottom  panel  for  both  daytime  and  nighttime  data. 

The  heavy  dark  line  at  the  zero  value  indicates  that  the  WWMCA  is 
equivalent  to  reference  analyses  (random  analyses  statistically  independent  of 
observations).  A  perfect  WWMCA  cloud  analysis  would  have  a  value  of  one. 
Negative  values  indicate  WWMCA  performs  worse  than  reference  analyses. 
Results  based  on  using  a  Cloudsat  cloud  mask  value  of  20  for  the  cloud/no¬ 
cloud  threshold. 
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